openai_compatible_server.md 24.6 KB
Newer Older
1
(openai-compatible-server)=
2

3
# OpenAI-Compatible Server
4

5
vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more! This functionality lets you serve models and interact with them using an HTTP client.
6

7
In your terminal, you can [install](../getting_started/installation.md) vLLM, then start the server with the [`vllm serve`](#vllm-serve) command. (You can also use our [Docker](#deployment-docker) image.)
8

9
```bash
10
vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
11
12
```

13
To call the server, in your preferred text editor, create a script that uses an HTTP client. Include any messages that you want to send to the model. Then run that script. Below is an example script using the [official OpenAI Python client](https://github.com/openai/openai-python).
14

15
16
17
18
19
20
21
22
```python
from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",
)

completion = client.chat.completions.create(
23
  model="NousResearch/Meta-Llama-3-8B-Instruct",
24
25
26
27
28
29
30
31
  messages=[
    {"role": "user", "content": "Hello!"}
  ]
)

print(completion.choices[0].message)
```

32
33
34
35
:::{tip}
vLLM supports some parameters that are not supported by OpenAI, `top_k` for example.
You can pass these parameters to vLLM using the OpenAI client in the `extra_body` parameter of your requests, i.e. `extra_body={"top_k": 50}` for `top_k`.
:::
36

37
38
:::{important}
By default, the server applies `generation_config.json` from the Hugging Face model repository if it exists. This means the default values of certain sampling parameters can be overridden by those recommended by the model creator.
39

40
41
To disable this behavior, please pass `--generation-config vllm` when launching the server.
:::
42

43
## Supported APIs
44

45
46
We currently support the following OpenAI APIs:

47
- [Completions API](#completions-api) (`/v1/completions`)
48
  - Only applicable to [text generation models](../models/generative_models.md) (`--task generate`).
49
  - *Note: `suffix` parameter is not supported.*
50
- [Chat Completions API](#chat-api) (`/v1/chat/completions`)
51
  - Only applicable to [text generation models](../models/generative_models.md) (`--task generate`) with a [chat template](#chat-template).
52
  - *Note: `parallel_tool_calls` and `user` parameters are ignored.*
53
- [Embeddings API](#embeddings-api) (`/v1/embeddings`)
54
  - Only applicable to [embedding models](../models/pooling_models.md) (`--task embed`).
55
56
- [Transcriptions API](#transcriptions-api) (`/v1/audio/transcriptions`)
  - Only applicable to Automatic Speech Recognition (ASR) models (OpenAI Whisper) (`--task generate`).
57

58
In addition, we have the following custom APIs:
59

60
61
- [Tokenizer API](#tokenizer-api) (`/tokenize`, `/detokenize`)
  - Applicable to any model with a tokenizer.
62
63
- [Pooling API](#pooling-api) (`/pooling`)
  - Applicable to all [pooling models](../models/pooling_models.md).
64
65
- [Classification API](#classification-api) (`/classify`)
  - Only applicable to [classification models](../models/pooling_models.md) (`--task classify`).
66
- [Score API](#score-api) (`/score`)
67
  - Applicable to embedding models and [cross-encoder models](../models/pooling_models.md) (`--task score`).
68
69
70
71
72
- [Re-rank API](#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)
  - Implements [Jina AI's v1 re-rank API](https://jina.ai/reranker/)
  - Also compatible with [Cohere's v1 & v2 re-rank APIs](https://docs.cohere.com/v2/reference/rerank)
  - Jina and Cohere's APIs are very similar; Jina's includes extra information in the rerank endpoint's response.
  - Only applicable to [cross-encoder models](../models/pooling_models.md) (`--task score`).
73

74
(chat-template)=
75

76
## Chat Template
77

78
79
80
In order for the language model to support chat protocol, vLLM requires the model to include
a chat template in its tokenizer configuration. The chat template is a Jinja2 template that
specifies how are roles, messages, and other chat-specific tokens are encoded in the input.
81

82
An example chat template for `NousResearch/Meta-Llama-3-8B-Instruct` can be found [here](https://github.com/meta-llama/llama3?tab=readme-ov-file#instruction-tuned-models)
83

84
85
86
87
Some models do not provide a chat template even though they are instruction/chat fine-tuned. For those model,
you can manually specify their chat template in the `--chat-template` parameter with the file path to the chat
template, or the template in string form. Without a chat template, the server will not be able to process chat
and all chat requests will error.
88
89

```bash
90
vllm serve <model> --chat-template ./path-to-chat-template.jinja
91
92
```

93
vLLM community provides a set of chat templates for popular models. You can find them under the <gh-dir:examples> directory.
94

95
96
With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies
both a `type` and a `text` field. An example is provided below:
97

98
99
100
101
102
```python
completion = client.chat.completions.create(
  model="NousResearch/Meta-Llama-3-8B-Instruct",
  messages=[
    {"role": "user", "content": [{"type": "text", "text": "Classify this sentiment: vLLM is wonderful!"}]}
103
  ]
104
)
105
106
```

107
Most chat templates for LLMs expect the `content` field to be a string, but there are some newer models like
108
109
110
111
`meta-llama/Llama-Guard-3-1B` that expect the content to be formatted according to the OpenAI schema in the
request. vLLM provides best-effort support to detect this automatically, which is logged as a string like
*"Detected the chat template content format to be..."*, and internally converts incoming requests to match
the detected format, which can be one of:
112

113
114
115
116
- `"string"`: A string.
  - Example: `"Hello world"`
- `"openai"`: A list of dictionaries, similar to OpenAI schema.
  - Example: `[{"type": "text", "text": "Hello world!"}]`
117

118
119
If the result is not what you expect, you can set the `--chat-template-content-format` CLI argument
to override which format to use.
120

121
## Extra Parameters
122

123
124
125
126
127
128
vLLM supports a set of parameters that are not part of the OpenAI API.
In order to use them, you can pass them as extra parameters in the OpenAI client.
Or directly merge them into the JSON payload if you are using HTTP call directly.

```python
completion = client.chat.completions.create(
129
  model="NousResearch/Meta-Llama-3-8B-Instruct",
130
131
132
133
134
135
136
137
138
  messages=[
    {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
  ],
  extra_body={
    "guided_choice": ["positive", "negative"]
  }
)
```

139
## Extra HTTP Headers
140

141
Only `X-Request-Id` HTTP request header is supported for now. It can be enabled
142
with `--enable-request-id-headers`.
143
144
145
146

> Note that enablement of the headers can impact performance significantly at high QPS
> rates. We recommend implementing HTTP headers at the router level (e.g. via Istio),
> rather than within the vLLM layer for this reason.
147
> See [this PR](https://github.com/vllm-project/vllm/pull/11529) for more details.
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170

```python
completion = client.chat.completions.create(
  model="NousResearch/Meta-Llama-3-8B-Instruct",
  messages=[
    {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
  ],
  extra_headers={
    "x-request-id": "sentiment-classification-00001",
  }
)
print(completion._request_id)

completion = client.completions.create(
  model="NousResearch/Meta-Llama-3-8B-Instruct",
  prompt="A robot may not injure a human being",
  extra_headers={
    "x-request-id": "completion-test",
  }
)
print(completion._request_id)
```

171
172
173
## CLI Reference

(vllm-serve)=
174

175
176
177
178
### `vllm serve`

The `vllm serve` command is used to launch the OpenAI-compatible server.

179
180
181
182
183
184
:::{tip}
The vast majority of command-line arguments are based on those for offline inference.

See [here](configuration-options) for some common options.
:::

185
:::{argparse}
186
187
188
:module: vllm.entrypoints.openai.cli_args
:func: create_parser_for_docs
:prog: vllm serve
189
:::
190
191
192
193
194
195
196
197
198
199
200

#### Configuration file

You can load CLI arguments via a [YAML](https://yaml.org/) config file.
The argument names must be the long form of those outlined [above](#vllm-serve).

For example:

```yaml
# config.yaml

201
model: meta-llama/Llama-3.1-8B-Instruct
202
203
204
205
206
207
208
209
host: "127.0.0.1"
port: 6379
uvicorn-log-level: "info"
```

To use the above config file:

```bash
210
vllm serve --config config.yaml
211
212
```

213
:::{note}
214
215
In case an argument is supplied simultaneously using command line and the config file, the value from the command line will take precedence.
The order of priorities is `command line > config file values > defaults`.
216
e.g. `vllm serve SOME_MODEL --config config.yaml`, SOME_MODEL takes precedence over `model` in config file.
217
:::
218
219
220
221

## API Reference

(completions-api)=
222

223
224
### Completions API

225
226
227
Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.

228
Code example: <gh-file:examples/online_serving/openai_completion_client.py>
229
230

#### Extra parameters
231

232
The following [sampling parameters](#sampling-params) are supported.
233

234
:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
235
236
237
:language: python
:start-after: begin-completion-sampling-params
:end-before: end-completion-sampling-params
238
:::
239
240
241

The following extra parameters are supported:

242
:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
243
244
245
:language: python
:start-after: begin-completion-extra-params
:end-before: end-completion-extra-params
246
:::
247

248
(chat-api)=
249

250
### Chat API
251

252
253
Our Chat API is compatible with [OpenAI's Chat Completions API](https://platform.openai.com/docs/api-reference/chat);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
254

255
256
We support both [Vision](https://platform.openai.com/docs/guides/vision)- and
[Audio](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in)-related parameters;
257
see our [Multimodal Inputs](#multimodal-inputs) guide for more information.
258
259
- *Note: `image_url.detail` parameter is not supported.*

260
Code example: <gh-file:examples/online_serving/openai_chat_completion_client.py>
261

262
#### Extra parameters
263

264
The following [sampling parameters](#sampling-params) are supported.
265

266
:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
267
268
269
:language: python
:start-after: begin-chat-completion-sampling-params
:end-before: end-chat-completion-sampling-params
270
:::
271
272
273

The following extra parameters are supported:

274
:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
275
276
277
:language: python
:start-after: begin-chat-completion-extra-params
:end-before: end-chat-completion-extra-params
278
:::
279

280
(embeddings-api)=
281

282
283
### Embeddings API

284
285
Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
286

287
If the model has a [chat template](#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](#chat-api))
288
289
which will be treated as a single prompt to the model.

290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
Code example: <gh-file:examples/online_serving/openai_embedding_client.py>

#### Multi-modal inputs

You can pass multi-modal inputs to embedding models by defining a custom chat template for the server
and passing a list of `messages` in the request. Refer to the examples below for illustration.

:::::{tab-set}
::::{tab-item} VLM2Vec

To serve the model:

```bash
vllm serve TIGER-Lab/VLM2Vec-Full --task embed \
  --trust-remote-code --max-model-len 4096 --chat-template examples/template_vlm2vec.jinja
```

:::{important}
Since VLM2Vec has the same model architecture as Phi-3.5-Vision, we have to explicitly pass `--task embed`
to run this model in embedding mode instead of text generation mode.

The custom chat template is completely different from the original one for this model,
and can be found here: <gh-file:examples/template_vlm2vec.jinja>
313
:::
314

315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
Since the request schema is not defined by OpenAI client, we post a request to the server using the lower-level `requests` library:

```python
import requests

image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"

response = requests.post(
    "http://localhost:8000/v1/embeddings",
    json={
        "model": "TIGER-Lab/VLM2Vec-Full",
        "messages": [{
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": image_url}},
                {"type": "text", "text": "Represent the given image."},
            ],
        }],
        "encoding_format": "float",
    },
)
response.raise_for_status()
response_json = response.json()
print("Embedding output:", response_json["data"][0]["embedding"])
```

::::

::::{tab-item} DSE-Qwen2-MRL

To serve the model:

```bash
vllm serve MrLight/dse-qwen2-2b-mrl-v1 --task embed \
  --trust-remote-code --max-model-len 8192 --chat-template examples/template_dse_qwen2_vl.jinja
```

:::{important}
Like with VLM2Vec, we have to explicitly pass `--task embed`.

Additionally, `MrLight/dse-qwen2-2b-mrl-v1` requires an EOS token for embeddings, which is handled
by a custom chat template: <gh-file:examples/template_dse_qwen2_vl.jinja>
:::

:::{important}
`MrLight/dse-qwen2-2b-mrl-v1` requires a placeholder image of the minimum image size for text query embeddings. See the full code
example below for details.
:::

::::

:::::

Full example: <gh-file:examples/online_serving/openai_chat_embedding_client_for_multimodal.py>
369

370
#### Extra parameters
371

372
The following [pooling parameters](#pooling-params) are supported.
373

374
:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
375
:language: python
376
377
:start-after: begin-embedding-pooling-params
:end-before: end-embedding-pooling-params
378
:::
379

380
The following extra parameters are supported by default:
381

382
:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
383
:language: python
384
385
:start-after: begin-embedding-extra-params
:end-before: end-embedding-extra-params
386
:::
387

388
For chat-like input (i.e. if `messages` is passed), these extra parameters are supported instead:
389

390
:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
391
392
393
:language: python
:start-after: begin-chat-embedding-extra-params
:end-before: end-chat-embedding-extra-params
394
:::
395

396
397
398
399
400
401
402
(transcriptions-api)=

### Transcriptions API

Our Transcriptions API is compatible with [OpenAI's Transcriptions API](https://platform.openai.com/docs/api-reference/audio/createTranscription);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.

403
404
405
406
:::{note}
To use the Transcriptions API, please install with extra audio dependencies using `pip install vllm[audio]`.
:::

407
Code example: <gh-file:examples/online_serving/openai_transcription_client.py>
408
409
<!-- TODO: api enforced limits + uploading audios -->

410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
#### Extra Parameters

The following [sampling parameters](#sampling-params) are supported.

:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
:language: python
:start-after: begin-transcription-sampling-params
:end-before: end-transcription-sampling-params
:::

The following extra parameters are supported:

:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
:language: python
:start-after: begin-transcription-extra-params
:end-before: end-transcription-extra-params
:::
427

428
(tokenizer-api)=
429

430
### Tokenizer API
431

432
Our Tokenizer API is a simple wrapper over [HuggingFace-style tokenizers](https://huggingface.co/docs/transformers/en/main_classes/tokenizer).
433
434
435
436
437
It consists of two endpoints:

- `/tokenize` corresponds to calling `tokenizer.encode()`.
- `/detokenize` corresponds to calling `tokenizer.decode()`.

438
(pooling-api)=
439

440
441
442
443
444
445
### Pooling API

Our Pooling API encodes input prompts using a [pooling model](../models/pooling_models.md) and returns the corresponding hidden states.

The input format is the same as [Embeddings API](#embeddings-api), but the output data can contain an arbitrary nested list, not just a 1-D list of floats.

446
Code example: <gh-file:examples/online_serving/openai_pooling_client.py>
447

448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
(classification-api)=

### Classification API

Our Classification API directly supports Hugging Face sequence-classification models such as [ai21labs/Jamba-tiny-reward-dev](https://huggingface.co/ai21labs/Jamba-tiny-reward-dev) and [jason9693/Qwen2.5-1.5B-apeach](https://huggingface.co/jason9693/Qwen2.5-1.5B-apeach).

We automatically wrap any other transformer via `as_classification_model()`, which pools on the last token, attaches a `RowParallelLinear` head, and applies a softmax to produce per-class probabilities.

Code example: <gh-file:examples/online_serving/openai_classification_client.py>

#### Example Requests

You can classify multiple texts by passing an array of strings:

Request:

```bash
curl -v "http://127.0.0.1:8000/classify" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jason9693/Qwen2.5-1.5B-apeach",
    "input": [
      "Loved the new café—coffee was great.",
      "This update broke everything. Frustrating."
    ]
  }'
```

Response:

```bash
{
  "id": "classify-7c87cac407b749a6935d8c7ce2a8fba2",
  "object": "list",
  "created": 1745383065,
  "model": "jason9693/Qwen2.5-1.5B-apeach",
  "data": [
    {
      "index": 0,
      "label": "Default",
      "probs": [
        0.565970778465271,
        0.4340292513370514
      ],
      "num_classes": 2
    },
    {
      "index": 1,
      "label": "Spoiled",
      "probs": [
        0.26448777318000793,
        0.7355121970176697
      ],
      "num_classes": 2
    }
  ],
  "usage": {
    "prompt_tokens": 20,
    "total_tokens": 20,
    "completion_tokens": 0,
    "prompt_tokens_details": null
  }
}
```

You can also pass a string directly to the `input` field:

Request:

```bash
curl -v "http://127.0.0.1:8000/classify" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jason9693/Qwen2.5-1.5B-apeach",
    "input": "Loved the new café—coffee was great."
  }'
```

Response:

```bash
{
  "id": "classify-9bf17f2847b046c7b2d5495f4b4f9682",
  "object": "list",
  "created": 1745383213,
  "model": "jason9693/Qwen2.5-1.5B-apeach",
  "data": [
    {
      "index": 0,
      "label": "Default",
      "probs": [
        0.565970778465271,
        0.4340292513370514
      ],
      "num_classes": 2
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "total_tokens": 10,
    "completion_tokens": 0,
    "prompt_tokens_details": null
  }
}
```

#### Extra parameters

The following [pooling parameters](#pooling-params) are supported.

:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
:language: python
:start-after: begin-classification-pooling-params
:end-before: end-classification-pooling-params
:::

The following extra parameters are supported:

:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
:language: python
:start-after: begin-classification-extra-params
:end-before: end-classification-extra-params
:::

572
(score-api)=
573

574
575
### Score API

576
Our Score API can apply a cross-encoder model or an embedding model to predict scores for sentence pairs. When using an embedding model the score corresponds to the cosine similarity between each embedding pair.
577
578
Usually, the score for a sentence pair refers to the similarity between two sentences, on a scale of 0 to 1.

579
You can find the documentation for cross encoder models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).
580

581
Code example: <gh-file:examples/online_serving/openai_cross_encoder_score.py>
582

583
584
585
586
587
#### Single inference

You can pass a string to both `text_1` and `text_2`, forming a single sentence pair.

Request:
588
589

```bash
590
591
592
593
594
595
596
597
598
599
curl -X 'POST' \
  'http://127.0.0.1:8000/score' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "BAAI/bge-reranker-v2-m3",
  "encoding_format": "float",
  "text_1": "What is the capital of France?",
  "text_2": "The capital of France is Paris."
}'
600
601
```

602
Response:
603

604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
```bash
{
  "id": "score-request-id",
  "object": "list",
  "created": 693447,
  "model": "BAAI/bge-reranker-v2-m3",
  "data": [
    {
      "index": 0,
      "object": "score",
      "score": 1
    }
  ],
  "usage": {}
}
619
620
```

621
#### Batch inference
622

623
624
625
You can pass a string to `text_1` and a list to `text_2`, forming multiple sentence pairs
where each pair is built from `text_1` and a string in `text_2`.
The total number of pairs is `len(text_2)`.
626

627
Request:
628

629
630
631
632
633
634
635
636
637
638
639
640
641
642
```bash
curl -X 'POST' \
  'http://127.0.0.1:8000/score' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "BAAI/bge-reranker-v2-m3",
  "text_1": "What is the capital of France?",
  "text_2": [
    "The capital of Brazil is Brasilia.",
    "The capital of France is Paris."
  ]
}'
```
643

644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
Response:

```bash
{
  "id": "score-request-id",
  "object": "list",
  "created": 693570,
  "model": "BAAI/bge-reranker-v2-m3",
  "data": [
    {
      "index": 0,
      "object": "score",
      "score": 0.001094818115234375
    },
    {
      "index": 1,
      "object": "score",
      "score": 1
    }
  ],
  "usage": {}
}
666
```
667

668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
You can pass a list to both `text_1` and `text_2`, forming multiple sentence pairs
where each pair is built from a string in `text_1` and the corresponding string in `text_2` (similar to `zip()`).
The total number of pairs is `len(text_2)`.

Request:

```bash
curl -X 'POST' \
  'http://127.0.0.1:8000/score' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "BAAI/bge-reranker-v2-m3",
  "encoding_format": "float",
  "text_1": [
    "What is the capital of Brazil?",
    "What is the capital of France?"
  ],
  "text_2": [
    "The capital of Brazil is Brasilia.",
    "The capital of France is Paris."
  ]
}'
```
692

693
Response:
694

695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
```bash
{
  "id": "score-request-id",
  "object": "list",
  "created": 693447,
  "model": "BAAI/bge-reranker-v2-m3",
  "data": [
    {
      "index": 0,
      "object": "score",
      "score": 1
    },
    {
      "index": 1,
      "object": "score",
      "score": 1
    }
  ],
  "usage": {}
}
```
716

717
#### Extra parameters
718

719
The following [pooling parameters](#pooling-params) are supported.
720

721
:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
722
723
724
:language: python
:start-after: begin-score-pooling-params
:end-before: end-score-pooling-params
725
:::
726

727
728
The following extra parameters are supported:

729
:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
730
731
732
:language: python
:start-after: begin-score-extra-params
:end-before: end-score-extra-params
733
:::
734
735
736
737
738

(rerank-api)=

### Re-rank API

739
Our Re-rank API can apply an embedding model or a cross-encoder model to predict relevant scores between a single query, and
740
741
742
each of a list of documents. Usually, the score for a sentence pair refers to the similarity between two sentences, on
a scale of 0 to 1.

743
You can find the documentation for cross encoder models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807

The rerank endpoints support popular re-rank models such as `BAAI/bge-reranker-base` and other models supporting the
`score` task. Additionally, `/rerank`, `/v1/rerank`, and `/v2/rerank`
endpoints are compatible with both [Jina AI's re-rank API interface](https://jina.ai/reranker/) and
[Cohere's re-rank API interface](https://docs.cohere.com/v2/reference/rerank) to ensure compatibility with
popular open-source tools.

Code example: <gh-file:examples/online_serving/jinaai_rerank_client.py>

#### Example Request

Note that the `top_n` request parameter is optional and will default to the length of the `documents` field.
Result documents will be sorted by relevance, and the `index` property can be used to determine original order.

Request:

```bash
curl -X 'POST' \
  'http://127.0.0.1:8000/v1/rerank' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "BAAI/bge-reranker-base",
  "query": "What is the capital of France?",
  "documents": [
    "The capital of Brazil is Brasilia.",
    "The capital of France is Paris.",
    "Horses and cows are both animals"
  ]
}'
```

Response:

```bash
{
  "id": "rerank-fae51b2b664d4ed38f5969b612edff77",
  "model": "BAAI/bge-reranker-base",
  "usage": {
    "total_tokens": 56
  },
  "results": [
    {
      "index": 1,
      "document": {
        "text": "The capital of France is Paris."
      },
      "relevance_score": 0.99853515625
    },
    {
      "index": 0,
      "document": {
        "text": "The capital of Brazil is Brasilia."
      },
      "relevance_score": 0.0005860328674316406
    }
  ]
}
```

#### Extra parameters

The following [pooling parameters](#pooling-params) are supported.

808
:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
809
810
811
:language: python
:start-after: begin-rerank-pooling-params
:end-before: end-rerank-pooling-params
812
:::
813
814
815

The following extra parameters are supported:

816
:::{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
817
818
819
:language: python
:start-after: begin-rerank-extra-params
:end-before: end-rerank-extra-params
820
:::