openai_compatible_server.md 27.3 KB
Newer Older
1
# OpenAI-Compatible Server
2

3
vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more! This functionality lets you serve models and interact with them using an HTTP client.
4

5
In your terminal, you can [install](../getting_started/installation/README.md) vLLM, then start the server with the [`vllm serve`](../configuration/serve_args.md) command. (You can also use our [Docker](../deployment/docker.md) image.)
6

7
```bash
Reid's avatar
Reid committed
8
9
10
vllm serve NousResearch/Meta-Llama-3-8B-Instruct \
  --dtype auto \
  --api-key token-abc123
11
12
```

13
To call the server, in your preferred text editor, create a script that uses an HTTP client. Include any messages that you want to send to the model. Then run that script. Below is an example script using the [official OpenAI Python client](https://github.com/openai/openai-python).
14

15
??? code
16

17
18
19
20
21
22
    ```python
    from openai import OpenAI
    client = OpenAI(
        base_url="http://localhost:8000/v1",
        api_key="token-abc123",
    )
23

24
25
26
27
28
29
30
31
32
    completion = client.chat.completions.create(
        model="NousResearch/Meta-Llama-3-8B-Instruct",
        messages=[
            {"role": "user", "content": "Hello!"}
        ]
    )

    print(completion.choices[0].message)
    ```
33

34
35
36
!!! tip
    vLLM supports some parameters that are not supported by OpenAI, `top_k` for example.
    You can pass these parameters to vLLM using the OpenAI client in the `extra_body` parameter of your requests, i.e. `extra_body={"top_k": 50}` for `top_k`.
37

38
!!! important
39
    By default, the server applies `generation_config.json` from the Hugging Face model repository if it exists. This means the default values of certain sampling parameters can be overridden by those recommended by the model creator.
40

41
    To disable this behavior, please pass `--generation-config vllm` when launching the server.
42

43
## Supported APIs
44

45
46
We currently support the following OpenAI APIs:

47
- [Completions API][completions-api] (`/v1/completions`)
48
    - Only applicable to [text generation models](../models/generative_models.md).
49
50
    - *Note: `suffix` parameter is not supported.*
- [Chat Completions API][chat-api] (`/v1/chat/completions`)
51
    - Only applicable to [text generation models](../models/generative_models.md) with a [chat template][chat-template].
52
53
    - *Note: `parallel_tool_calls` and `user` parameters are ignored.*
- [Embeddings API][embeddings-api] (`/v1/embeddings`)
54
    - Only applicable to [embedding models](../models/pooling_models.md).
55
- [Transcriptions API][transcriptions-api] (`/v1/audio/transcriptions`)
56
    - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#transcription).
57
- [Translation API][translations-api] (`/v1/audio/translations`)
58
    - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#transcription).
59

60
In addition, we have the following custom APIs:
61

62
63
64
65
66
- [Tokenizer API][tokenizer-api] (`/tokenize`, `/detokenize`)
    - Applicable to any model with a tokenizer.
- [Pooling API][pooling-api] (`/pooling`)
    - Applicable to all [pooling models](../models/pooling_models.md).
- [Classification API][classification-api] (`/classify`)
67
    - Only applicable to [classification models](../models/pooling_models.md).
68
- [Score API][score-api] (`/score`)
69
    - Applicable to [embedding models and cross-encoder models](../models/pooling_models.md).
70
71
72
73
- [Re-rank API][rerank-api] (`/rerank`, `/v1/rerank`, `/v2/rerank`)
    - Implements [Jina AI's v1 re-rank API](https://jina.ai/reranker/)
    - Also compatible with [Cohere's v1 & v2 re-rank APIs](https://docs.cohere.com/v2/reference/rerank)
    - Jina and Cohere's APIs are very similar; Jina's includes extra information in the rerank endpoint's response.
74
    - Only applicable to [cross-encoder models](../models/pooling_models.md).
75
76

[](){ #chat-template }
77

78
## Chat Template
79

80
81
82
In order for the language model to support chat protocol, vLLM requires the model to include
a chat template in its tokenizer configuration. The chat template is a Jinja2 template that
specifies how are roles, messages, and other chat-specific tokens are encoded in the input.
83

84
An example chat template for `NousResearch/Meta-Llama-3-8B-Instruct` can be found [here](https://github.com/meta-llama/llama3?tab=readme-ov-file#instruction-tuned-models)
85

86
87
88
89
Some models do not provide a chat template even though they are instruction/chat fine-tuned. For those model,
you can manually specify their chat template in the `--chat-template` parameter with the file path to the chat
template, or the template in string form. Without a chat template, the server will not be able to process chat
and all chat requests will error.
90
91

```bash
92
vllm serve <model> --chat-template ./path-to-chat-template.jinja
93
94
```

95
vLLM community provides a set of chat templates for popular models. You can find them under the <gh-dir:examples> directory.
96

97
98
With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies
both a `type` and a `text` field. An example is provided below:
99

100
101
```python
completion = client.chat.completions.create(
102
103
104
105
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": [{"type": "text", "text": "Classify this sentiment: vLLM is wonderful!"}]}
    ]
106
)
107
108
```

109
Most chat templates for LLMs expect the `content` field to be a string, but there are some newer models like
110
111
112
113
`meta-llama/Llama-Guard-3-1B` that expect the content to be formatted according to the OpenAI schema in the
request. vLLM provides best-effort support to detect this automatically, which is logged as a string like
*"Detected the chat template content format to be..."*, and internally converts incoming requests to match
the detected format, which can be one of:
114

115
- `"string"`: A string.
116
    - Example: `"Hello world"`
117
- `"openai"`: A list of dictionaries, similar to OpenAI schema.
118
    - Example: `[{"type": "text", "text": "Hello world!"}]`
119

120
121
If the result is not what you expect, you can set the `--chat-template-content-format` CLI argument
to override which format to use.
122

123
## Extra Parameters
124

125
126
127
128
129
130
vLLM supports a set of parameters that are not part of the OpenAI API.
In order to use them, you can pass them as extra parameters in the OpenAI client.
Or directly merge them into the JSON payload if you are using HTTP call directly.

```python
completion = client.chat.completions.create(
131
132
133
134
135
136
137
    model="NousResearch/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
    ],
    extra_body={
        "guided_choice": ["positive", "negative"]
    }
138
139
140
)
```

141
## Extra HTTP Headers
142

143
Only `X-Request-Id` HTTP request header is supported for now. It can be enabled
144
with `--enable-request-id-headers`.
145

146
??? code
147

148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
    ```python
    completion = client.chat.completions.create(
        model="NousResearch/Meta-Llama-3-8B-Instruct",
        messages=[
            {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
        ],
        extra_headers={
            "x-request-id": "sentiment-classification-00001",
        }
    )
    print(completion._request_id)

    completion = client.completions.create(
        model="NousResearch/Meta-Llama-3-8B-Instruct",
        prompt="A robot may not injure a human being",
        extra_headers={
            "x-request-id": "completion-test",
        }
    )
    print(completion._request_id)
    ```
169

170
171
## API Reference

172
[](){ #completions-api }
173

174
175
### Completions API

176
177
178
Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.

179
Code example: <gh-file:examples/online_serving/openai_completion_client.py>
180
181

#### Extra parameters
182

183
The following [sampling parameters][sampling-params] are supported.
184

185
??? code
186
187
188
189

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:completion-sampling-params"
    ```
190
191
192

The following extra parameters are supported:

193
??? code
194
195
196
197

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:completion-extra-params"
    ```
198

199
[](){ #chat-api }
200

201
### Chat API
202

203
204
Our Chat API is compatible with [OpenAI's Chat Completions API](https://platform.openai.com/docs/api-reference/chat);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
205

206
207
We support both [Vision](https://platform.openai.com/docs/guides/vision)- and
[Audio](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in)-related parameters;
208
see our [Multimodal Inputs](../features/multimodal_inputs.md) guide for more information.
209

210
211
- *Note: `image_url.detail` parameter is not supported.*

212
Code example: <gh-file:examples/online_serving/openai_chat_completion_client.py>
213

214
#### Extra parameters
215

216
The following [sampling parameters][sampling-params] are supported.
217

218
??? code
219
220
221
222

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-sampling-params"
    ```
223
224
225

The following extra parameters are supported:

226
??? code
227
228
229
230

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:chat-completion-extra-params"
    ```
231

232
[](){ #embeddings-api }
233

234
235
### Embeddings API

236
237
Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
238

239
If the model has a [chat template][chat-template], you can replace `inputs` with a list of `messages` (same schema as [Chat API][chat-api])
240
241
which will be treated as a single prompt to the model.

242
Code example: <gh-file:examples/online_serving/pooling/openai_embedding_client.py>
243
244
245
246
247
248

#### Multi-modal inputs

You can pass multi-modal inputs to embedding models by defining a custom chat template for the server
and passing a list of `messages` in the request. Refer to the examples below for illustration.

249
=== "VLM2Vec"
250

251
    To serve the model:
252

253
    ```bash
254
    vllm serve TIGER-Lab/VLM2Vec-Full --runner pooling \
Reid's avatar
Reid committed
255
256
257
      --trust-remote-code \
      --max-model-len 4096 \
      --chat-template examples/template_vlm2vec.jinja
258
    ```
259

260
    !!! important
261
        Since VLM2Vec has the same model architecture as Phi-3.5-Vision, we have to explicitly pass `--runner pooling`
262
        to run this model in embedding mode instead of text generation mode.
263

264
265
        The custom chat template is completely different from the original one for this model,
        and can be found here: <gh-file:examples/template_vlm2vec.jinja>
266

267
    Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:
268

269
    ??? code
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293

        ```python
        import requests

        image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"

        response = requests.post(
            "http://localhost:8000/v1/embeddings",
            json={
                "model": "TIGER-Lab/VLM2Vec-Full",
                "messages": [{
                    "role": "user",
                    "content": [
                        {"type": "image_url", "image_url": {"url": image_url}},
                        {"type": "text", "text": "Represent the given image."},
                    ],
                }],
                "encoding_format": "float",
            },
        )
        response.raise_for_status()
        response_json = response.json()
        print("Embedding output:", response_json["data"][0]["embedding"])
        ```
294

295
=== "DSE-Qwen2-MRL"
296

297
    To serve the model:
298

299
    ```bash
300
    vllm serve MrLight/dse-qwen2-2b-mrl-v1 --runner pooling \
Reid's avatar
Reid committed
301
302
303
      --trust-remote-code \
      --max-model-len 8192 \
      --chat-template examples/template_dse_qwen2_vl.jinja
304
    ```
305

306
    !!! important
307
        Like with VLM2Vec, we have to explicitly pass `--runner pooling`.
308

309
310
        Additionally, `MrLight/dse-qwen2-2b-mrl-v1` requires an EOS token for embeddings, which is handled
        by a custom chat template: <gh-file:examples/template_dse_qwen2_vl.jinja>
311

312
    !!! important
313
314
        `MrLight/dse-qwen2-2b-mrl-v1` requires a placeholder image of the minimum image size for text query embeddings. See the full code
        example below for details.
315

316
Full example: <gh-file:examples/online_serving/pooling/openai_chat_embedding_client_for_multimodal.py>
317

318
#### Extra parameters
319

320
The following [pooling parameters][pooling-params] are supported.
321

322
323
324
```python
--8<-- "vllm/entrypoints/openai/protocol.py:embedding-pooling-params"
```
325

326
The following extra parameters are supported by default:
327

328
??? code
329
330
331
332

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:embedding-extra-params"
    ```
333

334
For chat-like input (i.e. if `messages` is passed), these extra parameters are supported instead:
335

336
??? code
337
338
339
340

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:chat-embedding-extra-params"
    ```
341

342
[](){ #transcriptions-api }
343
344
345
346
347
348

### Transcriptions API

Our Transcriptions API is compatible with [OpenAI's Transcriptions API](https://platform.openai.com/docs/api-reference/audio/createTranscription);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.

349
350
!!! note
    To use the Transcriptions API, please install with extra audio dependencies using `pip install vllm[audio]`.
351

352
Code example: <gh-file:examples/online_serving/openai_transcription_client.py>
353
354
<!-- TODO: api enforced limits + uploading audios -->

355
356
357
358
359
#### API Enforced Limits

Set the maximum audio file size (in MB) that VLLM will accept, via the
`VLLM_MAX_AUDIO_CLIP_FILESIZE_MB` environment variable. Default is 25 MB.

360
361
#### Extra Parameters

362
The following [sampling parameters][sampling-params] are supported.
363

364
??? code
365
366
367
368

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:transcription-sampling-params"
    ```
369
370
371

The following extra parameters are supported:

372
??? code
373
374
375
376

    ```python
    --8<-- "vllm/entrypoints/openai/protocol.py:transcription-extra-params"
    ```
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
  
[](){ #translations-api }

### Translations API

Our Translation API is compatible with [OpenAI's Translations API](https://platform.openai.com/docs/api-reference/audio/createTranslation);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
Whisper models can translate audio from one of the 55 non-English supported languages into English.
Please mind that the popular `openai/whisper-large-v3-turbo` model does not support translating.

!!! note
    To use the Translation API, please install with extra audio dependencies using `pip install vllm[audio]`.

Code example: <gh-file:examples/online_serving/openai_translation_client.py>

#### Extra Parameters

The following [sampling parameters][sampling-params] are supported.

```python
--8<-- "vllm/entrypoints/openai/protocol.py:translation-sampling-params"
```

The following extra parameters are supported:

```python
--8<-- "vllm/entrypoints/openai/protocol.py:translation-extra-params"
```
405

406
[](){ #tokenizer-api }
407

408
### Tokenizer API
409

410
Our Tokenizer API is a simple wrapper over [HuggingFace-style tokenizers](https://huggingface.co/docs/transformers/en/main_classes/tokenizer).
411
412
413
414
415
It consists of two endpoints:

- `/tokenize` corresponds to calling `tokenizer.encode()`.
- `/detokenize` corresponds to calling `tokenizer.decode()`.

416
[](){ #pooling-api }
417

418
419
420
421
### Pooling API

Our Pooling API encodes input prompts using a [pooling model](../models/pooling_models.md) and returns the corresponding hidden states.

422
The input format is the same as [Embeddings API][embeddings-api], but the output data can contain an arbitrary nested list, not just a 1-D list of floats.
423

424
Code example: <gh-file:examples/online_serving/pooling/openai_pooling_client.py>
425

426
[](){ #classification-api }
427
428
429
430
431

### Classification API

Our Classification API directly supports Hugging Face sequence-classification models such as [ai21labs/Jamba-tiny-reward-dev](https://huggingface.co/ai21labs/Jamba-tiny-reward-dev) and [jason9693/Qwen2.5-1.5B-apeach](https://huggingface.co/jason9693/Qwen2.5-1.5B-apeach).

432
We automatically wrap any other transformer via `as_seq_cls_model()`, which pools on the last token, attaches a `RowParallelLinear` head, and applies a softmax to produce per-class probabilities.
433

434
Code example: <gh-file:examples/online_serving/pooling/openai_classification_client.py>
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451

#### Example Requests

You can classify multiple texts by passing an array of strings:

```bash
curl -v "http://127.0.0.1:8000/classify" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jason9693/Qwen2.5-1.5B-apeach",
    "input": [
      "Loved the new café—coffee was great.",
      "This update broke everything. Frustrating."
    ]
  }'
```

452
??? console "Response"
453

454
    ```json
455
    {
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
      "id": "classify-7c87cac407b749a6935d8c7ce2a8fba2",
      "object": "list",
      "created": 1745383065,
      "model": "jason9693/Qwen2.5-1.5B-apeach",
      "data": [
        {
          "index": 0,
          "label": "Default",
          "probs": [
            0.565970778465271,
            0.4340292513370514
          ],
          "num_classes": 2
        },
        {
          "index": 1,
          "label": "Spoiled",
          "probs": [
            0.26448777318000793,
            0.7355121970176697
          ],
          "num_classes": 2
        }
479
      ],
480
481
482
483
484
485
      "usage": {
        "prompt_tokens": 20,
        "total_tokens": 20,
        "completion_tokens": 0,
        "prompt_tokens_details": null
      }
486
    }
487
    ```
488
489
490
491
492
493
494
495
496
497
498
499

You can also pass a string directly to the `input` field:

```bash
curl -v "http://127.0.0.1:8000/classify" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jason9693/Qwen2.5-1.5B-apeach",
    "input": "Loved the new café—coffee was great."
  }'
```

500
??? console "Response"
501

502
    ```json
503
    {
504
505
506
507
508
509
510
511
512
513
514
515
516
517
      "id": "classify-9bf17f2847b046c7b2d5495f4b4f9682",
      "object": "list",
      "created": 1745383213,
      "model": "jason9693/Qwen2.5-1.5B-apeach",
      "data": [
        {
          "index": 0,
          "label": "Default",
          "probs": [
            0.565970778465271,
            0.4340292513370514
          ],
          "num_classes": 2
        }
518
      ],
519
520
521
522
523
524
      "usage": {
        "prompt_tokens": 10,
        "total_tokens": 10,
        "completion_tokens": 0,
        "prompt_tokens_details": null
      }
525
    }
526
    ```
527
528
529

#### Extra parameters

530
The following [pooling parameters][pooling-params] are supported.
531

532
533
534
```python
--8<-- "vllm/entrypoints/openai/protocol.py:classification-pooling-params"
```
535
536
537

The following extra parameters are supported:

538
539
540
```python
--8<-- "vllm/entrypoints/openai/protocol.py:classification-extra-params"
```
541

542
[](){ #score-api }
543

544
545
### Score API

546
Our Score API can apply a cross-encoder model or an embedding model to predict scores for sentence or multimodal pairs. When using an embedding model the score corresponds to the cosine similarity between each embedding pair.
547
548
Usually, the score for a sentence pair refers to the similarity between two sentences, on a scale of 0 to 1.

549
You can find the documentation for cross encoder models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).
550

551
Code example: <gh-file:examples/online_serving/openai_cross_encoder_score.py>
552

553
554
555
556
#### Single inference

You can pass a string to both `text_1` and `text_2`, forming a single sentence pair.

557
```bash
558
559
560
561
562
563
564
565
566
567
curl -X 'POST' \
  'http://127.0.0.1:8000/score' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "BAAI/bge-reranker-v2-m3",
  "encoding_format": "float",
  "text_1": "What is the capital of France?",
  "text_2": "The capital of France is Paris."
}'
568
569
```

570
??? console "Response"
571

572
    ```json
573
    {
574
575
576
577
578
579
580
581
582
583
584
585
      "id": "score-request-id",
      "object": "list",
      "created": 693447,
      "model": "BAAI/bge-reranker-v2-m3",
      "data": [
        {
          "index": 0,
          "object": "score",
          "score": 1
        }
      ],
      "usage": {}
586
    }
587
    ```
588

589
#### Batch inference
590

591
592
593
You can pass a string to `text_1` and a list to `text_2`, forming multiple sentence pairs
where each pair is built from `text_1` and a string in `text_2`.
The total number of pairs is `len(text_2)`.
594

595
??? console "Request"
596

597
598
599
600
601
602
603
604
605
606
607
608
609
610
    ```bash
    curl -X 'POST' \
      'http://127.0.0.1:8000/score' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "model": "BAAI/bge-reranker-v2-m3",
      "text_1": "What is the capital of France?",
      "text_2": [
        "The capital of Brazil is Brasilia.",
        "The capital of France is Paris."
      ]
    }'
    ```
611

612
??? console "Response"
613

614
    ```json
615
    {
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
      "id": "score-request-id",
      "object": "list",
      "created": 693570,
      "model": "BAAI/bge-reranker-v2-m3",
      "data": [
        {
          "index": 0,
          "object": "score",
          "score": 0.001094818115234375
        },
        {
          "index": 1,
          "object": "score",
          "score": 1
        }
      ],
      "usage": {}
633
    }
634
    ```
635

636
637
638
639
You can pass a list to both `text_1` and `text_2`, forming multiple sentence pairs
where each pair is built from a string in `text_1` and the corresponding string in `text_2` (similar to `zip()`).
The total number of pairs is `len(text_2)`.

640
??? console "Request"
641

642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
    ```bash
    curl -X 'POST' \
      'http://127.0.0.1:8000/score' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "model": "BAAI/bge-reranker-v2-m3",
      "encoding_format": "float",
      "text_1": [
        "What is the capital of Brazil?",
        "What is the capital of France?"
      ],
      "text_2": [
        "The capital of Brazil is Brasilia.",
        "The capital of France is Paris."
      ]
    }'
    ```
660

661
??? console "Response"
662

663
    ```json
664
    {
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
      "id": "score-request-id",
      "object": "list",
      "created": 693447,
      "model": "BAAI/bge-reranker-v2-m3",
      "data": [
        {
          "index": 0,
          "object": "score",
          "score": 1
        },
        {
          "index": 1,
          "object": "score",
          "score": 1
        }
      ],
      "usage": {}
682
    }
683
    ```
684

685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
#### Multi-modal inputs

You can pass multi-modal inputs to scoring models by passing `content` including a list of multi-modal input (image, etc.) in the request. Refer to the examples below for illustration.

=== "JinaVL-Reranker"

    To serve the model:

    ```bash
    vllm serve jinaai/jina-reranker-m0
    ```

    Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:

    ??? Code

        ```python
        import requests

        response = requests.post(
            "http://localhost:8000/v1/score",
            json={
                "model": "jinaai/jina-reranker-m0",
                "text_1": "slm markdown",
                "text_2": {
                  "content": [
                          {
                              "type": "image_url",
                              "image_url": {
                                  "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/handelsblatt-preview.png"
                              },
                          },
                          {
                              "type": "image_url",
                              "image_url": {
                                  "url": "https://raw.githubusercontent.com/jina-ai/multimodal-reranker-test/main/paper-11.png"
                              },
                          },
                      ]
                  }
                },
        )
        response.raise_for_status()
        response_json = response.json()
        print("Scoring output:", response_json["data"][0]["score"])
        print("Scoring output:", response_json["data"][1]["score"])
        ```
Full example: <gh-file:examples/online_serving/openai_cross_encoder_score_for_multimodal.py>

734
#### Extra parameters
735

736
The following [pooling parameters][pooling-params] are supported.
737

738
739
740
```python
--8<-- "vllm/entrypoints/openai/protocol.py:score-pooling-params"
```
741

742
743
The following extra parameters are supported:

744
745
746
```python
--8<-- "vllm/entrypoints/openai/protocol.py:score-extra-params"
```
747

748
[](){ #rerank-api }
749
750
751

### Re-rank API

752
Our Re-rank API can apply an embedding model or a cross-encoder model to predict relevant scores between a single query, and
753
each of a list of documents. Usually, the score for a sentence pair refers to the similarity between two sentences or multi-modal inputs (image, etc.), on a scale of 0 to 1.
754

755
You can find the documentation for cross encoder models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).
756
757
758
759
760
761
762

The rerank endpoints support popular re-rank models such as `BAAI/bge-reranker-base` and other models supporting the
`score` task. Additionally, `/rerank`, `/v1/rerank`, and `/v2/rerank`
endpoints are compatible with both [Jina AI's re-rank API interface](https://jina.ai/reranker/) and
[Cohere's re-rank API interface](https://docs.cohere.com/v2/reference/rerank) to ensure compatibility with
popular open-source tools.

763
Code example: <gh-file:examples/online_serving/pooling/jinaai_rerank_client.py>
764
765
766
767
768
769

#### Example Request

Note that the `top_n` request parameter is optional and will default to the length of the `documents` field.
Result documents will be sorted by relevance, and the `index` property can be used to determine original order.

770
??? console "Request"
771

772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
    ```bash
    curl -X 'POST' \
      'http://127.0.0.1:8000/v1/rerank' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "model": "BAAI/bge-reranker-base",
      "query": "What is the capital of France?",
      "documents": [
        "The capital of Brazil is Brasilia.",
        "The capital of France is Paris.",
        "Horses and cows are both animals"
      ]
    }'
    ```
787

788
??? console "Response"
789

790
    ```json
791
    {
792
793
794
795
      "id": "rerank-fae51b2b664d4ed38f5969b612edff77",
      "model": "BAAI/bge-reranker-base",
      "usage": {
        "total_tokens": 56
796
      },
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
      "results": [
        {
          "index": 1,
          "document": {
            "text": "The capital of France is Paris."
          },
          "relevance_score": 0.99853515625
        },
        {
          "index": 0,
          "document": {
            "text": "The capital of Brazil is Brasilia."
          },
          "relevance_score": 0.0005860328674316406
        }
      ]
813
    }
814
    ```
815
816
817

#### Extra parameters

818
The following [pooling parameters][pooling-params] are supported.
819

820
821
822
```python
--8<-- "vllm/entrypoints/openai/protocol.py:rerank-pooling-params"
```
823
824
825

The following extra parameters are supported:

826
827
828
```python
--8<-- "vllm/entrypoints/openai/protocol.py:rerank-extra-params"
```
829
830
831
832
833
834
835
836
837
838
839
840
841
842

## Ray Serve LLM

Ray Serve LLM enables scalable, production-grade serving of the vLLM engine. It integrates tightly with vLLM and extends it with features such as auto-scaling, load balancing, and back-pressure.

Key capabilities:

- Exposes an OpenAI-compatible HTTP API as well as a Pythonic API.
- Scales from a single GPU to a multi-node cluster without code changes.
- Provides observability and autoscaling policies through Ray dashboards and metrics.

The following example shows how to deploy a large model like DeepSeek R1 with Ray Serve LLM: <gh-file:examples/online_serving/ray_serve_deepseek.py>.

Learn more about Ray Serve LLM with the official [Ray Serve LLM documentation](https://docs.ray.io/en/latest/serve/llm/serving-llms.html).