openai_compatible_server.md 33.7 KB
Newer Older
1
# OpenAI-Compatible Server
2

3
vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more! This functionality lets you serve models and interact with them using an HTTP client.
4

5
In your terminal, you can [install](../getting_started/installation/README.md) vLLM, then start the server with the [`vllm serve`](../configuration/serve_args.md) command. (You can also use our [Docker](../deployment/docker.md) image.)
6

7
```bash
Reid's avatar
Reid committed
8
9
10
vllm serve NousResearch/Meta-Llama-3-8B-Instruct \
  --dtype auto \
  --api-key token-abc123
11
12
```

13
To call the server, in your preferred text editor, create a script that uses an HTTP client. Include any messages that you want to send to the model. Then run that script. Below is an example script using the [official OpenAI Python client](https://github.com/openai/openai-python).
14

15
??? code
16

17
18
19
20
21
22
    ```python
    from openai import OpenAI
    client = OpenAI(
        base_url="http://localhost:8000/v1",
        api_key="token-abc123",
    )
23

24
25
26
    completion = client.chat.completions.create(
        model="NousResearch/Meta-Llama-3-8B-Instruct",
        messages=[
27
28
            {"role": "user", "content": "Hello!"},
        ],
29
30
31
32
    )

    print(completion.choices[0].message)
    ```
33

34
35
36
!!! tip
    vLLM supports some parameters that are not supported by OpenAI, `top_k` for example.
    You can pass these parameters to vLLM using the OpenAI client in the `extra_body` parameter of your requests, i.e. `extra_body={"top_k": 50}` for `top_k`.
37

38
!!! important
39
    By default, the server applies `generation_config.json` from the Hugging Face model repository if it exists. This means the default values of certain sampling parameters can be overridden by those recommended by the model creator.
40

41
    To disable this behavior, please pass `--generation-config vllm` when launching the server.
42

43
## Supported APIs
44

45
46
We currently support the following OpenAI APIs:

47
- [Completions API](#completions-api) (`/v1/completions`)
48
    - Only applicable to [text generation models](../models/generative_models.md).
49
    - *Note: `suffix` parameter is not supported.*
50
51
- [Responses API](#responses-api) (`/v1/responses`)
    - Only applicable to [text generation models](../models/generative_models.md).
52
53
- [Chat Completions API](#chat-api) (`/v1/chat/completions`)
    - Only applicable to [text generation models](../models/generative_models.md) with a [chat template](../serving/openai_compatible_server.md#chat-template).
54
55
    - *Note: `user` parameter is ignored.*
    - *Note:* Setting the `parallel_tool_calls` parameter to `false` ensures vLLM only returns zero or one tool call per request. Setting it to `true` (the default) allows returning more than one tool call per request. There is no guarantee more than one tool call will be returned if this is set to `true`, as that behavior is model dependent and not all models are designed to support parallel tool calls.
56
- [Embeddings API](#embeddings-api) (`/v1/embeddings`)
57
    - Only applicable to [embedding models](../models/pooling_models.md).
58
- [Transcriptions API](#transcriptions-api) (`/v1/audio/transcriptions`)
59
    - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#transcription).
60
- [Translation API](#translations-api) (`/v1/audio/translations`)
61
    - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#transcription).
62

63
In addition, we have the following custom APIs:
64

65
- [Tokenizer API](#tokenizer-api) (`/tokenize`, `/detokenize`)
66
    - Applicable to any model with a tokenizer.
67
- [Pooling API](#pooling-api) (`/pooling`)
68
    - Applicable to all [pooling models](../models/pooling_models.md).
69
- [Classification API](#classification-api) (`/classify`)
70
    - Only applicable to [classification models](../models/pooling_models.md).
71
- [Score API](#score-api) (`/score`)
72
    - Applicable to [embedding models and cross-encoder models](../models/pooling_models.md).
73
- [Re-rank API](#re-rank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)
74
75
76
    - Implements [Jina AI's v1 re-rank API](https://jina.ai/reranker/)
    - Also compatible with [Cohere's v1 & v2 re-rank APIs](https://docs.cohere.com/v2/reference/rerank)
    - Jina and Cohere's APIs are very similar; Jina's includes extra information in the rerank endpoint's response.
77
    - Only applicable to [cross-encoder models](../models/pooling_models.md).
78

79
## Chat Template
80

81
82
In order for the language model to support chat protocol, vLLM requires the model to include
a chat template in its tokenizer configuration. The chat template is a Jinja2 template that
83
specifies how roles, messages, and other chat-specific tokens are encoded in the input.
84

85
An example chat template for `NousResearch/Meta-Llama-3-8B-Instruct` can be found [here](https://github.com/meta-llama/llama3?tab=readme-ov-file#instruction-tuned-models)
86

87
Some models do not provide a chat template even though they are instruction/chat fine-tuned. For those models,
88
89
90
you can manually specify their chat template in the `--chat-template` parameter with the file path to the chat
template, or the template in string form. Without a chat template, the server will not be able to process chat
and all chat requests will error.
91
92

```bash
93
vllm serve <model> --chat-template ./path-to-chat-template.jinja
94
95
```

96
vLLM community provides a set of chat templates for popular models. You can find them under the [examples](../../examples) directory.
97

98
99
With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies
both a `type` and a `text` field. An example is provided below:
100

101
102
```python
completion = client.chat.completions.create(
103
104
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[
105
106
107
108
109
110
111
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Classify this sentiment: vLLM is wonderful!"},
            ],
        },
    ],
112
)
113
114
```

115
Most chat templates for LLMs expect the `content` field to be a string, but there are some newer models like
116
117
118
119
`meta-llama/Llama-Guard-3-1B` that expect the content to be formatted according to the OpenAI schema in the
request. vLLM provides best-effort support to detect this automatically, which is logged as a string like
*"Detected the chat template content format to be..."*, and internally converts incoming requests to match
the detected format, which can be one of:
120

121
- `"string"`: A string.
122
    - Example: `"Hello world"`
123
- `"openai"`: A list of dictionaries, similar to OpenAI schema.
124
    - Example: `[{"type": "text", "text": "Hello world!"}]`
125

126
127
If the result is not what you expect, you can set the `--chat-template-content-format` CLI argument
to override which format to use.
128

129
## Extra Parameters
130

131
132
133
134
135
136
vLLM supports a set of parameters that are not part of the OpenAI API.
In order to use them, you can pass them as extra parameters in the OpenAI client.
Or directly merge them into the JSON payload if you are using HTTP call directly.

```python
completion = client.chat.completions.create(
137
138
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[
139
        {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"},
140
141
    ],
    extra_body={
142
143
        "structured_outputs": {"choice": ["positive", "negative"]},
    },
144
145
146
)
```

147
## Extra HTTP Headers
148

149
Only `X-Request-Id` HTTP request header is supported for now. It can be enabled
150
with `--enable-request-id-headers`.
151

152
??? code
153

154
155
156
157
    ```python
    completion = client.chat.completions.create(
        model="NousResearch/Meta-Llama-3-8B-Instruct",
        messages=[
158
            {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"},
159
160
161
        ],
        extra_headers={
            "x-request-id": "sentiment-classification-00001",
162
        },
163
164
165
166
167
168
169
170
    )
    print(completion._request_id)

    completion = client.completions.create(
        model="NousResearch/Meta-Llama-3-8B-Instruct",
        prompt="A robot may not injure a human being",
        extra_headers={
            "x-request-id": "completion-test",
171
        },
172
173
174
    )
    print(completion._request_id)
    ```
175

176
177
178
179
## API Reference

### Completions API

180
181
182
Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.

183
Code example: [examples/online_serving/openai_completion_client.py](../../examples/online_serving/openai_completion_client.py)
184
185

#### Extra parameters
186

187
The following [sampling parameters](../api/README.md#inference-parameters) are supported.
188

189
??? code
190
191
192
193

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:completion-sampling-params"
    ```
194
195
196

The following extra parameters are supported:

197
??? code
198
199
200
201

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:completion-extra-params"
    ```
202

203
### Chat API
204

205
206
Our Chat API is compatible with [OpenAI's Chat Completions API](https://platform.openai.com/docs/api-reference/chat);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
207

208
209
We support both [Vision](https://platform.openai.com/docs/guides/vision)- and
[Audio](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in)-related parameters;
210
see our [Multimodal Inputs](../features/multimodal_inputs.md) guide for more information.
211

212
213
- *Note: `image_url.detail` parameter is not supported.*

214
Code example: [examples/online_serving/openai_chat_completion_client.py](../../examples/online_serving/openai_chat_completion_client.py)
215

216
#### Extra parameters
217

218
The following [sampling parameters](../api/README.md#inference-parameters) are supported.
219

220
??? code
221
222
223
224

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-sampling-params"
    ```
225
226
227

The following extra parameters are supported:

228
??? code
229
230
231
232

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-extra-params"
    ```
233

234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
### Responses API

Our Responses API is compatible with [OpenAI's Responses API](https://platform.openai.com/docs/api-reference/responses);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.

Code example: [examples/online_serving/openai_responses_client_with_tools.py](../../examples/online_serving/openai_responses_client_with_tools.py)

#### Extra parameters

The following extra parameters in the request object are supported:

??? code

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:responses-extra-params"
    ```

The following extra parameters in the response object are supported:

??? code

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:responses-response-extra-params"
    ```

259
260
### Embeddings API

261
262
Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
263

264
Code example: [examples/pooling/embed/openai_embedding_client.py](../../examples/pooling/embed/openai_embedding_client.py)
265

266
If the model has a [chat template](../serving/openai_compatible_server.md#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](#chat-api))
267
which will be treated as a single prompt to the model. Here is a convenience function for calling the API while retaining OpenAI's type annotations:
268

269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
??? code

    ```python
    from openai import OpenAI
    from openai._types import NOT_GIVEN, NotGiven
    from openai.types.chat import ChatCompletionMessageParam
    from openai.types.create_embedding_response import CreateEmbeddingResponse

    def create_chat_embeddings(
        client: OpenAI,
        *,
        messages: list[ChatCompletionMessageParam],
        model: str,
        encoding_format: Union[Literal["base64", "float"], NotGiven] = NOT_GIVEN,
    ) -> CreateEmbeddingResponse:
        return client.post(
            "/embeddings",
            cast_to=CreateEmbeddingResponse,
            body={"messages": messages, "model": model, "encoding_format": encoding_format},
        )
    ```
290
291
292
293
294
295

#### Multi-modal inputs

You can pass multi-modal inputs to embedding models by defining a custom chat template for the server
and passing a list of `messages` in the request. Refer to the examples below for illustration.

296
=== "VLM2Vec"
297

298
    To serve the model:
299

300
    ```bash
301
    vllm serve TIGER-Lab/VLM2Vec-Full --runner pooling \
Reid's avatar
Reid committed
302
303
      --trust-remote-code \
      --max-model-len 4096 \
304
      --chat-template examples/template_vlm2vec_phi3v.jinja
305
    ```
306

307
    !!! important
308
        Since VLM2Vec has the same model architecture as Phi-3.5-Vision, we have to explicitly pass `--runner pooling`
309
        to run this model in embedding mode instead of text generation mode.
310

311
        The custom chat template is completely different from the original one for this model,
312
        and can be found here: [examples/template_vlm2vec_phi3v.jinja](../../examples/template_vlm2vec_phi3v.jinja)
313

314
    Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:
315

316
    ??? code
317
318

        ```python
319
320
321
322
323
        from openai import OpenAI
        client = OpenAI(
            base_url="http://localhost:8000/v1",
            api_key="EMPTY",
        )
324
        image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
325

326
327
328
329
330
        response = create_chat_embeddings(
            client,
            model="TIGER-Lab/VLM2Vec-Full",
            messages=[
                {
331
332
333
334
335
                    "role": "user",
                    "content": [
                        {"type": "image_url", "image_url": {"url": image_url}},
                        {"type": "text", "text": "Represent the given image."},
                    ],
336
337
338
                }
            ],
            encoding_format="float",
339
        )
340
341

        print("Image embedding output:", response.data[0].embedding)
342
        ```
343

344
=== "DSE-Qwen2-MRL"
345

346
    To serve the model:
347

348
    ```bash
349
    vllm serve MrLight/dse-qwen2-2b-mrl-v1 --runner pooling \
Reid's avatar
Reid committed
350
351
352
      --trust-remote-code \
      --max-model-len 8192 \
      --chat-template examples/template_dse_qwen2_vl.jinja
353
    ```
354

355
    !!! important
356
        Like with VLM2Vec, we have to explicitly pass `--runner pooling`.
357

358
        Additionally, `MrLight/dse-qwen2-2b-mrl-v1` requires an EOS token for embeddings, which is handled
359
        by a custom chat template: [examples/template_dse_qwen2_vl.jinja](../../examples/template_dse_qwen2_vl.jinja)
360

361
    !!! important
362
363
        `MrLight/dse-qwen2-2b-mrl-v1` requires a placeholder image of the minimum image size for text query embeddings. See the full code
        example below for details.
364

365
Full example: [examples/pooling/embed/vision_embedding_online.py](../../examples/pooling/embed/vision_embedding_online.py)
366

367
#### Extra parameters
368

369
The following [pooling parameters][vllm.PoolingParams] are supported.
370

371
```python
372
373
--8<-- "vllm/pooling_params.py:common-pooling-params"
--8<-- "vllm/pooling_params.py:embedding-pooling-params"
374
```
375

376
The following extra parameters are supported by default:
377

378
??? code
379
380

    ```python
381
    --8<-- "vllm/entrypoints/pooling/embed/protocol.py:embedding-extra-params"
382
    ```
383

384
For chat-like input (i.e. if `messages` is passed), these extra parameters are supported instead:
385

386
??? code
387
388

    ```python
389
    --8<-- "vllm/entrypoints/pooling/embed/protocol.py:chat-embedding-extra-params"
390
    ```
391

392
393
394
395
396
### Transcriptions API

Our Transcriptions API is compatible with [OpenAI's Transcriptions API](https://platform.openai.com/docs/api-reference/audio/createTranscription);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.

397
398
!!! note
    To use the Transcriptions API, please install with extra audio dependencies using `pip install vllm[audio]`.
399

400
Code example: [examples/online_serving/openai_transcription_client.py](../../examples/online_serving/openai_transcription_client.py)
401

402
403
404
405
406
#### API Enforced Limits

Set the maximum audio file size (in MB) that VLLM will accept, via the
`VLLM_MAX_AUDIO_CLIP_FILESIZE_MB` environment variable. Default is 25 MB.

407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
#### Uploading Audio Files

The Transcriptions API supports uploading audio files in various formats including FLAC, MP3, MP4, MPEG, MPGA, M4A, OGG, WAV, and WEBM.

**Using OpenAI Python Client:**

??? code

    ```python
    from openai import OpenAI

    client = OpenAI(
        base_url="http://localhost:8000/v1",
        api_key="token-abc123",
    )

    # Upload audio file from disk
    with open("audio.mp3", "rb") as audio_file:
        transcription = client.audio.transcriptions.create(
            model="openai/whisper-large-v3-turbo",
            file=audio_file,
            language="en",
429
            response_format="verbose_json",
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
        )

    print(transcription.text)
    ```

**Using curl with multipart/form-data:**

??? code

    ```bash
    curl -X POST "http://localhost:8000/v1/audio/transcriptions" \
      -H "Authorization: Bearer token-abc123" \
      -F "file=@audio.mp3" \
      -F "model=openai/whisper-large-v3-turbo" \
      -F "language=en" \
      -F "response_format=verbose_json"
    ```

**Supported Parameters:**

- `file`: The audio file to transcribe (required)
- `model`: The model to use for transcription (required)
- `language`: The language code (e.g., "en", "zh") (optional)
- `prompt`: Optional text to guide the transcription style (optional)
- `response_format`: Format of the response ("json", "text") (optional)
- `temperature`: Sampling temperature between 0 and 1 (optional)

For the complete list of supported parameters including sampling parameters and vLLM extensions, see the [protocol definitions](https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/openai/protocol.py#L2182).

**Response Format:**

For `verbose_json` response format:

??? code

    ```json
    {
      "text": "Hello, this is a transcription of the audio file.",
      "language": "en",
      "duration": 5.42,
      "segments": [
        {
          "id": 0,
          "seek": 0,
          "start": 0.0,
          "end": 2.5,
          "text": "Hello, this is a transcription",
          "tokens": [50364, 938, 428, 307, 275, 28347],
          "temperature": 0.0,
          "avg_logprob": -0.245,
          "compression_ratio": 1.235,
          "no_speech_prob": 0.012
        }
      ]
    }
    ```
486
Currently “verbose_json” response format doesn’t support avg_logprob, compression_ratio, no_speech_prob.
487

488
489
#### Extra Parameters

490
The following [sampling parameters](../api/README.md#inference-parameters) are supported.
491

492
??? code
493
494
495
496

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:transcription-sampling-params"
    ```
497
498
499

The following extra parameters are supported:

500
??? code
501
502
503
504

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:transcription-extra-params"
    ```
505

506
507
508
509
510
511
512
513
514
515
### Translations API

Our Translation API is compatible with [OpenAI's Translations API](https://platform.openai.com/docs/api-reference/audio/createTranslation);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
Whisper models can translate audio from one of the 55 non-English supported languages into English.
Please mind that the popular `openai/whisper-large-v3-turbo` model does not support translating.

!!! note
    To use the Translation API, please install with extra audio dependencies using `pip install vllm[audio]`.

516
Code example: [examples/online_serving/openai_translation_client.py](../../examples/online_serving/openai_translation_client.py)
517
518
519

#### Extra Parameters

520
The following [sampling parameters](../api/README.md#inference-parameters) are supported.
521
522
523
524
525
526
527
528
529
530

```python
--8<-- "vllm/entrypoints/openai/protocol.py:translation-sampling-params"
```

The following extra parameters are supported:

```python
--8<-- "vllm/entrypoints/openai/protocol.py:translation-extra-params"
```
531

532
### Tokenizer API
533

534
Our Tokenizer API is a simple wrapper over [HuggingFace-style tokenizers](https://huggingface.co/docs/transformers/en/main_classes/tokenizer).
535
536
537
538
539
It consists of two endpoints:

- `/tokenize` corresponds to calling `tokenizer.encode()`.
- `/detokenize` corresponds to calling `tokenizer.decode()`.

540
541
542
543
### Pooling API

Our Pooling API encodes input prompts using a [pooling model](../models/pooling_models.md) and returns the corresponding hidden states.

544
The input format is the same as [Embeddings API](#embeddings-api), but the output data can contain an arbitrary nested list, not just a 1-D list of floats.
545

546
Code example: [examples/pooling/pooling/openai_pooling_client.py](../../examples/pooling/pooling/openai_pooling_client.py)
547

548
549
550
551
### Classification API

Our Classification API directly supports Hugging Face sequence-classification models such as [ai21labs/Jamba-tiny-reward-dev](https://huggingface.co/ai21labs/Jamba-tiny-reward-dev) and [jason9693/Qwen2.5-1.5B-apeach](https://huggingface.co/jason9693/Qwen2.5-1.5B-apeach).

552
We automatically wrap any other transformer via `as_seq_cls_model()`, which pools on the last token, attaches a `RowParallelLinear` head, and applies a softmax to produce per-class probabilities.
553

554
Code example: [examples/pooling/classify/openai_classification_client.py](../../examples/pooling/classify/openai_classification_client.py)
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571

#### Example Requests

You can classify multiple texts by passing an array of strings:

```bash
curl -v "http://127.0.0.1:8000/classify" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jason9693/Qwen2.5-1.5B-apeach",
    "input": [
      "Loved the new café—coffee was great.",
      "This update broke everything. Frustrating."
    ]
  }'
```

572
??? console "Response"
573

574
    ```json
575
    {
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
      "id": "classify-7c87cac407b749a6935d8c7ce2a8fba2",
      "object": "list",
      "created": 1745383065,
      "model": "jason9693/Qwen2.5-1.5B-apeach",
      "data": [
        {
          "index": 0,
          "label": "Default",
          "probs": [
            0.565970778465271,
            0.4340292513370514
          ],
          "num_classes": 2
        },
        {
          "index": 1,
          "label": "Spoiled",
          "probs": [
            0.26448777318000793,
            0.7355121970176697
          ],
          "num_classes": 2
        }
599
      ],
600
601
602
603
604
605
      "usage": {
        "prompt_tokens": 20,
        "total_tokens": 20,
        "completion_tokens": 0,
        "prompt_tokens_details": null
      }
606
    }
607
    ```
608
609
610
611
612
613
614
615
616
617
618
619

You can also pass a string directly to the `input` field:

```bash
curl -v "http://127.0.0.1:8000/classify" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jason9693/Qwen2.5-1.5B-apeach",
    "input": "Loved the new café—coffee was great."
  }'
```

620
??? console "Response"
621

622
    ```json
623
    {
624
625
626
627
628
629
630
631
632
633
634
635
636
637
      "id": "classify-9bf17f2847b046c7b2d5495f4b4f9682",
      "object": "list",
      "created": 1745383213,
      "model": "jason9693/Qwen2.5-1.5B-apeach",
      "data": [
        {
          "index": 0,
          "label": "Default",
          "probs": [
            0.565970778465271,
            0.4340292513370514
          ],
          "num_classes": 2
        }
638
      ],
639
640
641
642
643
644
      "usage": {
        "prompt_tokens": 10,
        "total_tokens": 10,
        "completion_tokens": 0,
        "prompt_tokens_details": null
      }
645
    }
646
    ```
647
648
649

#### Extra parameters

650
The following [pooling parameters][vllm.PoolingParams] are supported.
651

652
```python
653
654
--8<-- "vllm/pooling_params.py:common-pooling-params"
--8<-- "vllm/pooling_params.py:classification-pooling-params"
655
```
656
657
658

The following extra parameters are supported:

659
```python
660
--8<-- "vllm/entrypoints/pooling/classify/protocol.py:classification-extra-params"
661
```
662

663
664
### Score API

665
Our Score API can apply a cross-encoder model or an embedding model to predict scores for sentence or multimodal pairs. When using an embedding model the score corresponds to the cosine similarity between each embedding pair.
666
667
Usually, the score for a sentence pair refers to the similarity between two sentences, on a scale of 0 to 1.

668
You can find the documentation for cross encoder models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).
669

670
Code example: [examples/pooling/score/score_api_online.py](../../examples/pooling/score/score_api_online.py)
671

672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
#### Score Template

Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](#chat-template)).

Score templates are supported for **cross-encoder** models only. If you are using an **embedding** model for scoring, vLLM does not apply a score template.

Like chat templates, the score template receives a `messages` list. For scoring, each message has a `role` attribute—either `"query"` or `"document"`. For the usual kind of point-wise cross-encoder, you can expect exactly two messages: one query and one document. To access the query and document content, use Jinja's `selectattr` filter:

- **Query**: `{{ (messages | selectattr("role", "eq", "query") | first).content }}`
- **Document**: `{{ (messages | selectattr("role", "eq", "document") | first).content }}`

This approach is more robust than index-based access (`messages[0]`, `messages[1]`) because it selects messages by their semantic role. It also avoids assumptions about message ordering if additional message types are added to `messages` in the future.

Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../examples/pooling/score/template/nemotron-rerank.jinja)

687
688
689
690
#### Single inference

You can pass a string to both `text_1` and `text_2`, forming a single sentence pair.

691
```bash
692
693
694
695
696
697
698
699
700
701
curl -X 'POST' \
  'http://127.0.0.1:8000/score' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "BAAI/bge-reranker-v2-m3",
  "encoding_format": "float",
  "text_1": "What is the capital of France?",
  "text_2": "The capital of France is Paris."
}'
702
703
```

704
??? console "Response"
705

706
    ```json
707
    {
708
709
710
711
712
713
714
715
716
717
718
719
      "id": "score-request-id",
      "object": "list",
      "created": 693447,
      "model": "BAAI/bge-reranker-v2-m3",
      "data": [
        {
          "index": 0,
          "object": "score",
          "score": 1
        }
      ],
      "usage": {}
720
    }
721
    ```
722

723
#### Batch inference
724

725
726
727
You can pass a string to `text_1` and a list to `text_2`, forming multiple sentence pairs
where each pair is built from `text_1` and a string in `text_2`.
The total number of pairs is `len(text_2)`.
728

729
??? console "Request"
730

731
732
733
734
735
736
737
738
739
740
741
742
743
744
    ```bash
    curl -X 'POST' \
      'http://127.0.0.1:8000/score' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "model": "BAAI/bge-reranker-v2-m3",
      "text_1": "What is the capital of France?",
      "text_2": [
        "The capital of Brazil is Brasilia.",
        "The capital of France is Paris."
      ]
    }'
    ```
745

746
??? console "Response"
747

748
    ```json
749
    {
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
      "id": "score-request-id",
      "object": "list",
      "created": 693570,
      "model": "BAAI/bge-reranker-v2-m3",
      "data": [
        {
          "index": 0,
          "object": "score",
          "score": 0.001094818115234375
        },
        {
          "index": 1,
          "object": "score",
          "score": 1
        }
      ],
      "usage": {}
767
    }
768
    ```
769

770
771
772
773
You can pass a list to both `text_1` and `text_2`, forming multiple sentence pairs
where each pair is built from a string in `text_1` and the corresponding string in `text_2` (similar to `zip()`).
The total number of pairs is `len(text_2)`.

774
??? console "Request"
775

776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
    ```bash
    curl -X 'POST' \
      'http://127.0.0.1:8000/score' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "model": "BAAI/bge-reranker-v2-m3",
      "encoding_format": "float",
      "text_1": [
        "What is the capital of Brazil?",
        "What is the capital of France?"
      ],
      "text_2": [
        "The capital of Brazil is Brasilia.",
        "The capital of France is Paris."
      ]
    }'
    ```
794

795
??? console "Response"
796

797
    ```json
798
    {
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
      "id": "score-request-id",
      "object": "list",
      "created": 693447,
      "model": "BAAI/bge-reranker-v2-m3",
      "data": [
        {
          "index": 0,
          "object": "score",
          "score": 1
        },
        {
          "index": 1,
          "object": "score",
          "score": 1
        }
      ],
      "usage": {}
816
    }
817
    ```
818

819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
#### Multi-modal inputs

You can pass multi-modal inputs to scoring models by passing `content` including a list of multi-modal input (image, etc.) in the request. Refer to the examples below for illustration.

=== "JinaVL-Reranker"

    To serve the model:

    ```bash
    vllm serve jinaai/jina-reranker-m0
    ```

    Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:

    ??? Code

        ```python
        import requests

        response = requests.post(
            "http://localhost:8000/v1/score",
            json={
                "model": "jinaai/jina-reranker-m0",
                "text_1": "slm markdown",
                "text_2": {
844
845
846
847
848
849
850
851
852
853
854
855
856
857
                    "content": [
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"
                            },
                        },
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/paper-11.png"
                            },
                        },
                    ],
858
                },
859
            },
860
861
862
863
864
865
        )
        response.raise_for_status()
        response_json = response.json()
        print("Scoring output:", response_json["data"][0]["score"])
        print("Scoring output:", response_json["data"][1]["score"])
        ```
866
867
868
869
Full example:

- [examples/pooling/score/vision_score_api_online.py](../../examples/pooling/score/vision_score_api_online.py)
- examples/pooling/score/vision_rerank_api_online.py](../../examples/pooling/score/vision_rerank_api_online.py)
870

871
#### Extra parameters
872

873
The following [pooling parameters][vllm.PoolingParams] are supported.
874

875
```python
876
877
--8<-- "vllm/pooling_params.py:common-pooling-params"
--8<-- "vllm/pooling_params.py:classification-pooling-params"
878
```
879

880
881
The following extra parameters are supported:

882
```python
883
--8<-- "vllm/entrypoints/pooling/score/protocol.py:score-extra-params"
884
```
885
886
887

### Re-rank API

888
Our Re-rank API can apply an embedding model or a cross-encoder model to predict relevant scores between a single query, and
889
each of a list of documents. Usually, the score for a sentence pair refers to the similarity between two sentences or multi-modal inputs (image, etc.), on a scale of 0 to 1.
890

891
You can find the documentation for cross encoder models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).
892
893
894
895
896
897
898

The rerank endpoints support popular re-rank models such as `BAAI/bge-reranker-base` and other models supporting the
`score` task. Additionally, `/rerank`, `/v1/rerank`, and `/v2/rerank`
endpoints are compatible with both [Jina AI's re-rank API interface](https://jina.ai/reranker/) and
[Cohere's re-rank API interface](https://docs.cohere.com/v2/reference/rerank) to ensure compatibility with
popular open-source tools.

899
Code example: [examples/pooling/score/rerank_api_online.py](../../examples/pooling/score/rerank_api_online.py)
900
901
902
903
904
905

#### Example Request

Note that the `top_n` request parameter is optional and will default to the length of the `documents` field.
Result documents will be sorted by relevance, and the `index` property can be used to determine original order.

906
??? console "Request"
907

908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
    ```bash
    curl -X 'POST' \
      'http://127.0.0.1:8000/v1/rerank' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "model": "BAAI/bge-reranker-base",
      "query": "What is the capital of France?",
      "documents": [
        "The capital of Brazil is Brasilia.",
        "The capital of France is Paris.",
        "Horses and cows are both animals"
      ]
    }'
    ```
923

924
??? console "Response"
925

926
    ```json
927
    {
928
929
930
931
      "id": "rerank-fae51b2b664d4ed38f5969b612edff77",
      "model": "BAAI/bge-reranker-base",
      "usage": {
        "total_tokens": 56
932
      },
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
      "results": [
        {
          "index": 1,
          "document": {
            "text": "The capital of France is Paris."
          },
          "relevance_score": 0.99853515625
        },
        {
          "index": 0,
          "document": {
            "text": "The capital of Brazil is Brasilia."
          },
          "relevance_score": 0.0005860328674316406
        }
      ]
949
    }
950
    ```
951
952
953

#### Extra parameters

954
The following [pooling parameters][vllm.PoolingParams] are supported.
955

956
```python
957
958
--8<-- "vllm/pooling_params.py:common-pooling-params"
--8<-- "vllm/pooling_params.py:classification-pooling-params"
959
```
960
961
962

The following extra parameters are supported:

963
```python
964
--8<-- "vllm/entrypoints/pooling/score/protocol.py:rerank-extra-params"
965
```
966
967
968
969
970
971
972
973
974
975
976

## Ray Serve LLM

Ray Serve LLM enables scalable, production-grade serving of the vLLM engine. It integrates tightly with vLLM and extends it with features such as auto-scaling, load balancing, and back-pressure.

Key capabilities:

- Exposes an OpenAI-compatible HTTP API as well as a Pythonic API.
- Scales from a single GPU to a multi-node cluster without code changes.
- Provides observability and autoscaling policies through Ray dashboards and metrics.

977
The following example shows how to deploy a large model like DeepSeek R1 with Ray Serve LLM: [examples/online_serving/ray_serve_deepseek.py](../../examples/online_serving/ray_serve_deepseek.py).
978
979

Learn more about Ray Serve LLM with the official [Ray Serve LLM documentation](https://docs.ray.io/en/latest/serve/llm/serving-llms.html).