openai_compatible_server.md 27.3 KB
Newer Older
1
# OpenAI-Compatible Server
2

3
vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more! This functionality lets you serve models and interact with them using an HTTP client.
4

5
In your terminal, you can [install](../getting_started/installation/README.md) vLLM, then start the server with the [`vllm serve`](../configuration/serve_args.md) command. (You can also use our [Docker](../deployment/docker.md) image.)
6

7
```bash
Reid's avatar
Reid committed
8
9
10
vllm serve NousResearch/Meta-Llama-3-8B-Instruct \
  --dtype auto \
  --api-key token-abc123
11
12
```

13
To call the server, in your preferred text editor, create a script that uses an HTTP client. Include any messages that you want to send to the model. Then run that script. Below is an example script using the [official OpenAI Python client](https://github.com/openai/openai-python).
14

15
??? code
16

17
18
19
20
21
22
    ```python
    from openai import OpenAI
    client = OpenAI(
        base_url="http://localhost:8000/v1",
        api_key="token-abc123",
    )
23

24
25
26
27
28
29
30
31
32
    completion = client.chat.completions.create(
        model="NousResearch/Meta-Llama-3-8B-Instruct",
        messages=[
            {"role": "user", "content": "Hello!"}
        ]
    )

    print(completion.choices[0].message)
    ```
33

34
35
36
!!! tip
    vLLM supports some parameters that are not supported by OpenAI, `top_k` for example.
    You can pass these parameters to vLLM using the OpenAI client in the `extra_body` parameter of your requests, i.e. `extra_body={"top_k": 50}` for `top_k`.
37

38
!!! important
39
    By default, the server applies `generation_config.json` from the Hugging Face model repository if it exists. This means the default values of certain sampling parameters can be overridden by those recommended by the model creator.
40

41
    To disable this behavior, please pass `--generation-config vllm` when launching the server.
42

43
## Supported APIs
44

45
46
We currently support the following OpenAI APIs:

47
- [Completions API][completions-api] (`/v1/completions`)
48
    - Only applicable to [text generation models](../models/generative_models.md).
49
50
    - *Note: `suffix` parameter is not supported.*
- [Chat Completions API][chat-api] (`/v1/chat/completions`)
51
    - Only applicable to [text generation models](../models/generative_models.md) with a [chat template][chat-template].
52
53
    - *Note: `parallel_tool_calls` and `user` parameters are ignored.*
- [Embeddings API][embeddings-api] (`/v1/embeddings`)
54
    - Only applicable to [embedding models](../models/pooling_models.md).
55
- [Transcriptions API][transcriptions-api] (`/v1/audio/transcriptions`)
56
    - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#transcription).
57
- [Translation API][translations-api] (`/v1/audio/translations`)
58
    - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#transcription).
59

60
In addition, we have the following custom APIs:
61

62
63
64
65
66
- [Tokenizer API][tokenizer-api] (`/tokenize`, `/detokenize`)
    - Applicable to any model with a tokenizer.
- [Pooling API][pooling-api] (`/pooling`)
    - Applicable to all [pooling models](../models/pooling_models.md).
- [Classification API][classification-api] (`/classify`)
67
    - Only applicable to [classification models](../models/pooling_models.md).
68
- [Score API][score-api] (`/score`)
69
    - Applicable to [embedding models and cross-encoder models](../models/pooling_models.md).
70
71
72
73
- [Re-rank API][rerank-api] (`/rerank`, `/v1/rerank`, `/v2/rerank`)
    - Implements [Jina AI's v1 re-rank API](https://jina.ai/reranker/)
    - Also compatible with [Cohere's v1 & v2 re-rank APIs](https://docs.cohere.com/v2/reference/rerank)
    - Jina and Cohere's APIs are very similar; Jina's includes extra information in the rerank endpoint's response.
74
    - Only applicable to [cross-encoder models](../models/pooling_models.md).
75
76

[](){ #chat-template }
77

78
## Chat Template
79

80
81
82
In order for the language model to support chat protocol, vLLM requires the model to include
a chat template in its tokenizer configuration. The chat template is a Jinja2 template that
specifies how are roles, messages, and other chat-specific tokens are encoded in the input.
83

84
An example chat template for `NousResearch/Meta-Llama-3-8B-Instruct` can be found [here](https://github.com/meta-llama/llama3?tab=readme-ov-file#instruction-tuned-models)
85

86
87
88
89
Some models do not provide a chat template even though they are instruction/chat fine-tuned. For those model,
you can manually specify their chat template in the `--chat-template` parameter with the file path to the chat
template, or the template in string form. Without a chat template, the server will not be able to process chat
and all chat requests will error.
90
91

```bash
92
vllm serve <model> --chat-template ./path-to-chat-template.jinja
93
94
```

95
vLLM community provides a set of chat templates for popular models. You can find them under the <gh-dir:examples> directory.
96

97
98
With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies
both a `type` and a `text` field. An example is provided below:
99

100
101
```python
completion = client.chat.completions.create(
102
103
104
105
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": [{"type": "text", "text": "Classify this sentiment: vLLM is wonderful!"}]}
    ]
106
)
107
108
```

109
Most chat templates for LLMs expect the `content` field to be a string, but there are some newer models like
110
111
112
113
`meta-llama/Llama-Guard-3-1B` that expect the content to be formatted according to the OpenAI schema in the
request. vLLM provides best-effort support to detect this automatically, which is logged as a string like
*"Detected the chat template content format to be..."*, and internally converts incoming requests to match
the detected format, which can be one of:
114

115
- `"string"`: A string.
116
    - Example: `"Hello world"`
117
- `"openai"`: A list of dictionaries, similar to OpenAI schema.
118
    - Example: `[{"type": "text", "text": "Hello world!"}]`
119

120
121
If the result is not what you expect, you can set the `--chat-template-content-format` CLI argument
to override which format to use.
122

123
## Extra Parameters
124

125
126
127
128
129
130
vLLM supports a set of parameters that are not part of the OpenAI API.
In order to use them, you can pass them as extra parameters in the OpenAI client.
Or directly merge them into the JSON payload if you are using HTTP call directly.

```python
completion = client.chat.completions.create(
131
132
133
134
135
136
137
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
    ],
    extra_body={
        "guided_choice": ["positive", "negative"]
    }
138
139
140
)
```

141
## Extra HTTP Headers
142

143
Only `X-Request-Id` HTTP request header is supported for now. It can be enabled
144
with `--enable-request-id-headers`.
145

146
??? code
147

148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
    ```python
    completion = client.chat.completions.create(
        model="NousResearch/Meta-Llama-3-8B-Instruct",
        messages=[
            {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
        ],
        extra_headers={
            "x-request-id": "sentiment-classification-00001",
        }
    )
    print(completion._request_id)

    completion = client.completions.create(
        model="NousResearch/Meta-Llama-3-8B-Instruct",
        prompt="A robot may not injure a human being",
        extra_headers={
            "x-request-id": "completion-test",
        }
    )
    print(completion._request_id)
    ```
169

170
171
## API Reference

172
[](){ #completions-api }
173

174
175
### Completions API

176
177
178
Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.

179
Code example: <gh-file:examples/online_serving/openai_completion_client.py>
180
181

#### Extra parameters
182

183
The following [sampling parameters][sampling-params] are supported.
184

185
??? code
186
187
188
189

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:completion-sampling-params"
    ```
190
191
192

The following extra parameters are supported:

193
??? code
194
195
196
197

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:completion-extra-params"
    ```
198

199
[](){ #chat-api }
200

201
### Chat API
202

203
204
Our Chat API is compatible with [OpenAI's Chat Completions API](https://platform.openai.com/docs/api-reference/chat);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
205

206
207
We support both [Vision](https://platform.openai.com/docs/guides/vision)- and
[Audio](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in)-related parameters;
208
see our [Multimodal Inputs](../features/multimodal_inputs.md) guide for more information.
209
210
- *Note: `image_url.detail` parameter is not supported.*

211
Code example: <gh-file:examples/online_serving/openai_chat_completion_client.py>
212

213
#### Extra parameters
214

215
The following [sampling parameters][sampling-params] are supported.
216

217
??? code
218
219
220
221

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-sampling-params"
    ```
222
223
224

The following extra parameters are supported:

225
??? code
226
227
228
229

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-extra-params"
    ```
230

231
[](){ #embeddings-api }
232

233
234
### Embeddings API

235
236
Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
237

238
If the model has a [chat template][chat-template], you can replace `inputs` with a list of `messages` (same schema as [Chat API][chat-api])
239
240
which will be treated as a single prompt to the model.

241
242
243
244
245
246
247
Code example: <gh-file:examples/online_serving/openai_embedding_client.py>

#### Multi-modal inputs

You can pass multi-modal inputs to embedding models by defining a custom chat template for the server
and passing a list of `messages` in the request. Refer to the examples below for illustration.

248
=== "VLM2Vec"
249

250
    To serve the model:
251

252
    ```bash
253
    vllm serve TIGER-Lab/VLM2Vec-Full --runner pooling \
Reid's avatar
Reid committed
254
255
256
      --trust-remote-code \
      --max-model-len 4096 \
      --chat-template examples/template_vlm2vec.jinja
257
    ```
258

259
    !!! important
260
        Since VLM2Vec has the same model architecture as Phi-3.5-Vision, we have to explicitly pass `--runner pooling`
261
        to run this model in embedding mode instead of text generation mode.
262

263
264
        The custom chat template is completely different from the original one for this model,
        and can be found here: <gh-file:examples/template_vlm2vec.jinja>
265

266
    Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:
267

268
    ??? code
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292

        ```python
        import requests

        image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"

        response = requests.post(
            "http://localhost:8000/v1/embeddings",
            json={
                "model": "TIGER-Lab/VLM2Vec-Full",
                "messages": [{
                    "role": "user",
                    "content": [
                        {"type": "image_url", "image_url": {"url": image_url}},
                        {"type": "text", "text": "Represent the given image."},
                    ],
                }],
                "encoding_format": "float",
            },
        )
        response.raise_for_status()
        response_json = response.json()
        print("Embedding output:", response_json["data"][0]["embedding"])
        ```
293

294
=== "DSE-Qwen2-MRL"
295

296
    To serve the model:
297

298
    ```bash
299
    vllm serve MrLight/dse-qwen2-2b-mrl-v1 --runner pooling \
Reid's avatar
Reid committed
300
301
302
      --trust-remote-code \
      --max-model-len 8192 \
      --chat-template examples/template_dse_qwen2_vl.jinja
303
    ```
304

305
    !!! important
306
        Like with VLM2Vec, we have to explicitly pass `--runner pooling`.
307

308
309
        Additionally, `MrLight/dse-qwen2-2b-mrl-v1` requires an EOS token for embeddings, which is handled
        by a custom chat template: <gh-file:examples/template_dse_qwen2_vl.jinja>
310

311
    !!! important
312
313
        `MrLight/dse-qwen2-2b-mrl-v1` requires a placeholder image of the minimum image size for text query embeddings. See the full code
        example below for details.
314
315

Full example: <gh-file:examples/online_serving/openai_chat_embedding_client_for_multimodal.py>
316

317
#### Extra parameters
318

319
The following [pooling parameters][pooling-params] are supported.
320

321
322
323
```python
--8<-- "vllm/entrypoints/openai/protocol.py:embedding-pooling-params"
```
324

325
The following extra parameters are supported by default:
326

327
??? code
328
329
330
331

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:embedding-extra-params"
    ```
332

333
For chat-like input (i.e. if `messages` is passed), these extra parameters are supported instead:
334

335
??? code
336
337
338
339

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:chat-embedding-extra-params"
    ```
340

341
[](){ #transcriptions-api }
342
343
344
345
346
347

### Transcriptions API

Our Transcriptions API is compatible with [OpenAI's Transcriptions API](https://platform.openai.com/docs/api-reference/audio/createTranscription);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.

348
349
!!! note
    To use the Transcriptions API, please install with extra audio dependencies using `pip install vllm[audio]`.
350

351
Code example: <gh-file:examples/online_serving/openai_transcription_client.py>
352
353
<!-- TODO: api enforced limits + uploading audios -->

354
355
356
357
358
#### API Enforced Limits

Set the maximum audio file size (in MB) that VLLM will accept, via the
`VLLM_MAX_AUDIO_CLIP_FILESIZE_MB` environment variable. Default is 25 MB.

359
360
#### Extra Parameters

361
The following [sampling parameters][sampling-params] are supported.
362

363
??? code
364
365
366
367

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:transcription-sampling-params"
    ```
368
369
370

The following extra parameters are supported:

371
??? code
372
373
374
375

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:transcription-extra-params"
    ```
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
  
[](){ #translations-api }

### Translations API

Our Translation API is compatible with [OpenAI's Translations API](https://platform.openai.com/docs/api-reference/audio/createTranslation);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
Whisper models can translate audio from one of the 55 non-English supported languages into English.
Please mind that the popular `openai/whisper-large-v3-turbo` model does not support translating.

!!! note
    To use the Translation API, please install with extra audio dependencies using `pip install vllm[audio]`.

Code example: <gh-file:examples/online_serving/openai_translation_client.py>

#### Extra Parameters

The following [sampling parameters][sampling-params] are supported.

```python
--8<-- "vllm/entrypoints/openai/protocol.py:translation-sampling-params"
```

The following extra parameters are supported:

```python
--8<-- "vllm/entrypoints/openai/protocol.py:translation-extra-params"
```
404

405
[](){ #tokenizer-api }
406

407
### Tokenizer API
408

409
Our Tokenizer API is a simple wrapper over [HuggingFace-style tokenizers](https://huggingface.co/docs/transformers/en/main_classes/tokenizer).
410
411
412
413
414
It consists of two endpoints:

- `/tokenize` corresponds to calling `tokenizer.encode()`.
- `/detokenize` corresponds to calling `tokenizer.decode()`.

415
[](){ #pooling-api }
416

417
418
419
420
### Pooling API

Our Pooling API encodes input prompts using a [pooling model](../models/pooling_models.md) and returns the corresponding hidden states.

421
The input format is the same as [Embeddings API][embeddings-api], but the output data can contain an arbitrary nested list, not just a 1-D list of floats.
422

423
Code example: <gh-file:examples/online_serving/openai_pooling_client.py>
424

425
[](){ #classification-api }
426
427
428
429
430

### Classification API

Our Classification API directly supports Hugging Face sequence-classification models such as [ai21labs/Jamba-tiny-reward-dev](https://huggingface.co/ai21labs/Jamba-tiny-reward-dev) and [jason9693/Qwen2.5-1.5B-apeach](https://huggingface.co/jason9693/Qwen2.5-1.5B-apeach).

431
We automatically wrap any other transformer via `as_seq_cls_model()`, which pools on the last token, attaches a `RowParallelLinear` head, and applies a softmax to produce per-class probabilities.
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450

Code example: <gh-file:examples/online_serving/openai_classification_client.py>

#### Example Requests

You can classify multiple texts by passing an array of strings:

```bash
curl -v "http://127.0.0.1:8000/classify" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jason9693/Qwen2.5-1.5B-apeach",
    "input": [
      "Loved the new café—coffee was great.",
      "This update broke everything. Frustrating."
    ]
  }'
```

451
??? console "Response"
452

453
    ```json
454
    {
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
      "id": "classify-7c87cac407b749a6935d8c7ce2a8fba2",
      "object": "list",
      "created": 1745383065,
      "model": "jason9693/Qwen2.5-1.5B-apeach",
      "data": [
        {
          "index": 0,
          "label": "Default",
          "probs": [
            0.565970778465271,
            0.4340292513370514
          ],
          "num_classes": 2
        },
        {
          "index": 1,
          "label": "Spoiled",
          "probs": [
            0.26448777318000793,
            0.7355121970176697
          ],
          "num_classes": 2
        }
478
      ],
479
480
481
482
483
484
      "usage": {
        "prompt_tokens": 20,
        "total_tokens": 20,
        "completion_tokens": 0,
        "prompt_tokens_details": null
      }
485
    }
486
    ```
487
488
489
490
491
492
493
494
495
496
497
498

You can also pass a string directly to the `input` field:

```bash
curl -v "http://127.0.0.1:8000/classify" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jason9693/Qwen2.5-1.5B-apeach",
    "input": "Loved the new café—coffee was great."
  }'
```

499
??? console "Response"
500

501
    ```json
502
    {
503
504
505
506
507
508
509
510
511
512
513
514
515
516
      "id": "classify-9bf17f2847b046c7b2d5495f4b4f9682",
      "object": "list",
      "created": 1745383213,
      "model": "jason9693/Qwen2.5-1.5B-apeach",
      "data": [
        {
          "index": 0,
          "label": "Default",
          "probs": [
            0.565970778465271,
            0.4340292513370514
          ],
          "num_classes": 2
        }
517
      ],
518
519
520
521
522
523
      "usage": {
        "prompt_tokens": 10,
        "total_tokens": 10,
        "completion_tokens": 0,
        "prompt_tokens_details": null
      }
524
    }
525
    ```
526
527
528

#### Extra parameters

529
The following [pooling parameters][pooling-params] are supported.
530

531
532
533
```python
--8<-- "vllm/entrypoints/openai/protocol.py:classification-pooling-params"
```
534
535
536

The following extra parameters are supported:

537
538
539
```python
--8<-- "vllm/entrypoints/openai/protocol.py:classification-extra-params"
```
540

541
[](){ #score-api }
542

543
544
### Score API

545
Our Score API can apply a cross-encoder model or an embedding model to predict scores for sentence or multimodal pairs. When using an embedding model the score corresponds to the cosine similarity between each embedding pair.
546
547
Usually, the score for a sentence pair refers to the similarity between two sentences, on a scale of 0 to 1.

548
You can find the documentation for cross encoder models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).
549

550
Code example: <gh-file:examples/online_serving/openai_cross_encoder_score.py>
551

552
553
554
555
#### Single inference

You can pass a string to both `text_1` and `text_2`, forming a single sentence pair.

556
```bash
557
558
559
560
561
562
563
564
565
566
curl -X 'POST' \
  'http://127.0.0.1:8000/score' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "BAAI/bge-reranker-v2-m3",
  "encoding_format": "float",
  "text_1": "What is the capital of France?",
  "text_2": "The capital of France is Paris."
}'
567
568
```

569
??? console "Response"
570

571
    ```json
572
    {
573
574
575
576
577
578
579
580
581
582
583
584
      "id": "score-request-id",
      "object": "list",
      "created": 693447,
      "model": "BAAI/bge-reranker-v2-m3",
      "data": [
        {
          "index": 0,
          "object": "score",
          "score": 1
        }
      ],
      "usage": {}
585
    }
586
    ```
587

588
#### Batch inference
589

590
591
592
You can pass a string to `text_1` and a list to `text_2`, forming multiple sentence pairs
where each pair is built from `text_1` and a string in `text_2`.
The total number of pairs is `len(text_2)`.
593

594
??? console "Request"
595

596
597
598
599
600
601
602
603
604
605
606
607
608
609
    ```bash
    curl -X 'POST' \
      'http://127.0.0.1:8000/score' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "model": "BAAI/bge-reranker-v2-m3",
      "text_1": "What is the capital of France?",
      "text_2": [
        "The capital of Brazil is Brasilia.",
        "The capital of France is Paris."
      ]
    }'
    ```
610

611
??? console "Response"
612

613
    ```json
614
    {
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
      "id": "score-request-id",
      "object": "list",
      "created": 693570,
      "model": "BAAI/bge-reranker-v2-m3",
      "data": [
        {
          "index": 0,
          "object": "score",
          "score": 0.001094818115234375
        },
        {
          "index": 1,
          "object": "score",
          "score": 1
        }
      ],
      "usage": {}
632
    }
633
    ```
634

635
636
637
638
You can pass a list to both `text_1` and `text_2`, forming multiple sentence pairs
where each pair is built from a string in `text_1` and the corresponding string in `text_2` (similar to `zip()`).
The total number of pairs is `len(text_2)`.

639
??? console "Request"
640

641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
    ```bash
    curl -X 'POST' \
      'http://127.0.0.1:8000/score' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "model": "BAAI/bge-reranker-v2-m3",
      "encoding_format": "float",
      "text_1": [
        "What is the capital of Brazil?",
        "What is the capital of France?"
      ],
      "text_2": [
        "The capital of Brazil is Brasilia.",
        "The capital of France is Paris."
      ]
    }'
    ```
659

660
??? console "Response"
661

662
    ```json
663
    {
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
      "id": "score-request-id",
      "object": "list",
      "created": 693447,
      "model": "BAAI/bge-reranker-v2-m3",
      "data": [
        {
          "index": 0,
          "object": "score",
          "score": 1
        },
        {
          "index": 1,
          "object": "score",
          "score": 1
        }
      ],
      "usage": {}
681
    }
682
    ```
683

684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
#### Multi-modal inputs

You can pass multi-modal inputs to scoring models by passing `content` including a list of multi-modal input (image, etc.) in the request. Refer to the examples below for illustration.

=== "JinaVL-Reranker"

    To serve the model:

    ```bash
    vllm serve jinaai/jina-reranker-m0
    ```

    Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:

    ??? Code

        ```python
        import requests

        response = requests.post(
            "http://localhost:8000/v1/score",
            json={
                "model": "jinaai/jina-reranker-m0",
                "text_1": "slm markdown",
                "text_2": {
                  "content": [
                          {
                              "type": "image_url",
                              "image_url": {
                                  "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"
                              },
                          },
                          {
                              "type": "image_url",
                              "image_url": {
                                  "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/paper-11.png"
                              },
                          },
                      ]
                  }
                },
        )
        response.raise_for_status()
        response_json = response.json()
        print("Scoring output:", response_json["data"][0]["score"])
        print("Scoring output:", response_json["data"][1]["score"])
        ```
Full example: <gh-file:examples/online_serving/openai_cross_encoder_score_for_multimodal.py>

733
#### Extra parameters
734

735
The following [pooling parameters][pooling-params] are supported.
736

737
738
739
```python
--8<-- "vllm/entrypoints/openai/protocol.py:score-pooling-params"
```
740

741
742
The following extra parameters are supported:

743
744
745
```python
--8<-- "vllm/entrypoints/openai/protocol.py:score-extra-params"
```
746

747
[](){ #rerank-api }
748
749
750

### Re-rank API

751
Our Re-rank API can apply an embedding model or a cross-encoder model to predict relevant scores between a single query, and
752
each of a list of documents. Usually, the score for a sentence pair refers to the similarity between two sentences or multi-modal inputs (image, etc.), on a scale of 0 to 1.
753

754
You can find the documentation for cross encoder models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).
755
756
757
758
759
760
761
762
763
764
765
766
767
768

The rerank endpoints support popular re-rank models such as `BAAI/bge-reranker-base` and other models supporting the
`score` task. Additionally, `/rerank`, `/v1/rerank`, and `/v2/rerank`
endpoints are compatible with both [Jina AI's re-rank API interface](https://jina.ai/reranker/) and
[Cohere's re-rank API interface](https://docs.cohere.com/v2/reference/rerank) to ensure compatibility with
popular open-source tools.

Code example: <gh-file:examples/online_serving/jinaai_rerank_client.py>

#### Example Request

Note that the `top_n` request parameter is optional and will default to the length of the `documents` field.
Result documents will be sorted by relevance, and the `index` property can be used to determine original order.

769
??? console "Request"
770

771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
    ```bash
    curl -X 'POST' \
      'http://127.0.0.1:8000/v1/rerank' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "model": "BAAI/bge-reranker-base",
      "query": "What is the capital of France?",
      "documents": [
        "The capital of Brazil is Brasilia.",
        "The capital of France is Paris.",
        "Horses and cows are both animals"
      ]
    }'
    ```
786

787
??? console "Response"
788

789
    ```json
790
    {
791
792
793
794
      "id": "rerank-fae51b2b664d4ed38f5969b612edff77",
      "model": "BAAI/bge-reranker-base",
      "usage": {
        "total_tokens": 56
795
      },
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
      "results": [
        {
          "index": 1,
          "document": {
            "text": "The capital of France is Paris."
          },
          "relevance_score": 0.99853515625
        },
        {
          "index": 0,
          "document": {
            "text": "The capital of Brazil is Brasilia."
          },
          "relevance_score": 0.0005860328674316406
        }
      ]
812
    }
813
    ```
814
815
816

#### Extra parameters

817
The following [pooling parameters][pooling-params] are supported.
818

819
820
821
```python
--8<-- "vllm/entrypoints/openai/protocol.py:rerank-pooling-params"
```
822
823
824

The following extra parameters are supported:

825
826
827
```python
--8<-- "vllm/entrypoints/openai/protocol.py:rerank-extra-params"
```
828
829
830
831
832
833
834
835
836
837
838
839
840
841

## Ray Serve LLM

Ray Serve LLM enables scalable, production-grade serving of the vLLM engine. It integrates tightly with vLLM and extends it with features such as auto-scaling, load balancing, and back-pressure.

Key capabilities:

- Exposes an OpenAI-compatible HTTP API as well as a Pythonic API.
- Scales from a single GPU to a multi-node cluster without code changes.
- Provides observability and autoscaling policies through Ray dashboards and metrics.

The following example shows how to deploy a large model like DeepSeek R1 with Ray Serve LLM: <gh-file:examples/online_serving/ray_serve_deepseek.py>.

Learn more about Ray Serve LLM with the official [Ray Serve LLM documentation](https://docs.ray.io/en/latest/serve/llm/serving-llms.html).