openai_compatible_server.md 14 KB
Newer Older
1
(openai-compatible-server)=
2

3
# OpenAI-Compatible Server
4

5
6
7
vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more!

You can start the server via the [`vllm serve`](#vllm-serve) command, or through [Docker](#deployment-docker):
8

9
```bash
10
vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
11
12
```

13
To call the server, you can use the [official OpenAI Python client](https://github.com/openai/openai-python), or any other HTTP client.
14

15
16
17
18
19
20
21
22
```python
from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",
)

completion = client.chat.completions.create(
23
  model="NousResearch/Meta-Llama-3-8B-Instruct",
24
25
26
27
28
29
30
31
  messages=[
    {"role": "user", "content": "Hello!"}
  ]
)

print(completion.choices[0].message)
```

32
## Supported APIs
33

34
35
We currently support the following OpenAI APIs:

36
- [Completions API](#completions-api) (`/v1/completions`)
37
  - Only applicable to [text generation models](../models/generative_models.md) (`--task generate`).
38
  - *Note: `suffix` parameter is not supported.*
39
- [Chat Completions API](#chat-api) (`/v1/chat/completions`)
40
  - Only applicable to [text generation models](../models/generative_models.md) (`--task generate`) with a [chat template](#chat-template).
41
  - *Note: `parallel_tool_calls` and `user` parameters are ignored.*
42
- [Embeddings API](#embeddings-api) (`/v1/embeddings`)
43
  - Only applicable to [embedding models](../models/pooling_models.md) (`--task embed`).
44

45
In addition, we have the following custom APIs:
46

47
48
- [Tokenizer API](#tokenizer-api) (`/tokenize`, `/detokenize`)
  - Applicable to any model with a tokenizer.
49
50
- [Pooling API](#pooling-api) (`/pooling`)
  - Applicable to all [pooling models](../models/pooling_models.md).
51
- [Score API](#score-api) (`/score`)
52
  - Only applicable to [cross-encoder models](../models/pooling_models.md) (`--task score`).
53

54
(chat-template)=
55

56
## Chat Template
57

58
59
60
In order for the language model to support chat protocol, vLLM requires the model to include
a chat template in its tokenizer configuration. The chat template is a Jinja2 template that
specifies how are roles, messages, and other chat-specific tokens are encoded in the input.
61

62
An example chat template for `NousResearch/Meta-Llama-3-8B-Instruct` can be found [here](https://github.com/meta-llama/llama3?tab=readme-ov-file#instruction-tuned-models)
63

64
65
66
67
Some models do not provide a chat template even though they are instruction/chat fine-tuned. For those model,
you can manually specify their chat template in the `--chat-template` parameter with the file path to the chat
template, or the template in string form. Without a chat template, the server will not be able to process chat
and all chat requests will error.
68
69

```bash
70
vllm serve <model> --chat-template ./path-to-chat-template.jinja
71
72
```

73
vLLM community provides a set of chat templates for popular models. You can find them under the <gh-dir:examples> directory.
74

75
76
With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies
both a `type` and a `text` field. An example is provided below:
77

78
79
80
81
82
```python
completion = client.chat.completions.create(
  model="NousResearch/Meta-Llama-3-8B-Instruct",
  messages=[
    {"role": "user", "content": [{"type": "text", "text": "Classify this sentiment: vLLM is wonderful!"}]}
83
  ]
84
)
85
86
```

87
Most chat templates for LLMs expect the `content` field to be a string, but there are some newer models like
88
89
90
91
`meta-llama/Llama-Guard-3-1B` that expect the content to be formatted according to the OpenAI schema in the
request. vLLM provides best-effort support to detect this automatically, which is logged as a string like
*"Detected the chat template content format to be..."*, and internally converts incoming requests to match
the detected format, which can be one of:
92

93
94
95
96
- `"string"`: A string.
  - Example: `"Hello world"`
- `"openai"`: A list of dictionaries, similar to OpenAI schema.
  - Example: `[{"type": "text", "text": "Hello world!"}]`
97

98
99
If the result is not what you expect, you can set the `--chat-template-content-format` CLI argument
to override which format to use.
100

101
## Extra Parameters
102

103
104
105
106
107
108
vLLM supports a set of parameters that are not part of the OpenAI API.
In order to use them, you can pass them as extra parameters in the OpenAI client.
Or directly merge them into the JSON payload if you are using HTTP call directly.

```python
completion = client.chat.completions.create(
109
  model="NousResearch/Meta-Llama-3-8B-Instruct",
110
111
112
113
114
115
116
117
118
  messages=[
    {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
  ],
  extra_body={
    "guided_choice": ["positive", "negative"]
  }
)
```

119
## Extra HTTP Headers
120

121
Only `X-Request-Id` HTTP request header is supported for now. It can be enabled
122
with `--enable-request-id-headers`.
123
124
125
126

> Note that enablement of the headers can impact performance significantly at high QPS
> rates. We recommend implementing HTTP headers at the router level (e.g. via Istio),
> rather than within the vLLM layer for this reason.
127
> See [this PR](https://github.com/vllm-project/vllm/pull/11529) for more details.
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150

```python
completion = client.chat.completions.create(
  model="NousResearch/Meta-Llama-3-8B-Instruct",
  messages=[
    {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
  ],
  extra_headers={
    "x-request-id": "sentiment-classification-00001",
  }
)
print(completion._request_id)

completion = client.completions.create(
  model="NousResearch/Meta-Llama-3-8B-Instruct",
  prompt="A robot may not injure a human being",
  extra_headers={
    "x-request-id": "completion-test",
  }
)
print(completion._request_id)
```

151
152
153
## CLI Reference

(vllm-serve)=
154

155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
### `vllm serve`

The `vllm serve` command is used to launch the OpenAI-compatible server.

```{argparse}
:module: vllm.entrypoints.openai.cli_args
:func: create_parser_for_docs
:prog: vllm serve
```

#### Configuration file

You can load CLI arguments via a [YAML](https://yaml.org/) config file.
The argument names must be the long form of those outlined [above](#vllm-serve).

For example:

```yaml
# config.yaml

host: "127.0.0.1"
port: 6379
uvicorn-log-level: "info"
```

To use the above config file:

```bash
183
vllm serve SOME_MODEL --config config.yaml
184
185
186
187
188
189
190
191
192
193
```

```{note}
In case an argument is supplied simultaneously using command line and the config file, the value from the command line will take precedence.
The order of priorities is `command line > config file values > defaults`.
```

## API Reference

(completions-api)=
194

195
196
### Completions API

197
198
199
Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.

200
Code example: <gh-file:examples/online_serving/openai_completion_client.py>
201
202

#### Extra parameters
203

204
The following [sampling parameters](#sampling-params) are supported.
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219

```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
:language: python
:start-after: begin-completion-sampling-params
:end-before: end-completion-sampling-params
```

The following extra parameters are supported:

```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
:language: python
:start-after: begin-completion-extra-params
:end-before: end-completion-extra-params
```

220
(chat-api)=
221

222
### Chat API
223

224
225
Our Chat API is compatible with [OpenAI's Chat Completions API](https://platform.openai.com/docs/api-reference/chat);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
226

227
228
We support both [Vision](https://platform.openai.com/docs/guides/vision)- and
[Audio](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in)-related parameters;
229
see our [Multimodal Inputs](#multimodal-inputs) guide for more information.
230
231
- *Note: `image_url.detail` parameter is not supported.*

232
Code example: <gh-file:examples/online_serving/openai_chat_completion_client.py>
233

234
#### Extra parameters
235

236
The following [sampling parameters](#sampling-params) are supported.
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251

```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
:language: python
:start-after: begin-chat-completion-sampling-params
:end-before: end-chat-completion-sampling-params
```

The following extra parameters are supported:

```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
:language: python
:start-after: begin-chat-completion-extra-params
:end-before: end-chat-completion-extra-params
```

252
(embeddings-api)=
253

254
255
### Embeddings API

256
257
Our Embeddings API is compatible with [OpenAI's Embeddings API](https://platform.openai.com/docs/api-reference/embeddings);
you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it.
258

259
If the model has a [chat template](#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](#chat-api))
260
261
262
which will be treated as a single prompt to the model.

```{tip}
263
This enables multi-modal inputs to be passed to embedding models, see [this page](#multimodal-inputs) for details.
264
265
```

266
Code example: <gh-file:examples/online_serving/openai_embedding_client.py>
267

268
#### Extra parameters
269

270
The following [pooling parameters](#pooling-params) are supported.
271
272
273

```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
:language: python
274
275
:start-after: begin-embedding-pooling-params
:end-before: end-embedding-pooling-params
276
277
```

278
The following extra parameters are supported by default:
279
280
281

```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
:language: python
282
283
:start-after: begin-embedding-extra-params
:end-before: end-embedding-extra-params
284
285
```

286
For chat-like input (i.e. if `messages` is passed), these extra parameters are supported instead:
287

288
289
290
291
292
```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
:language: python
:start-after: begin-chat-embedding-extra-params
:end-before: end-chat-embedding-extra-params
```
293

294
(tokenizer-api)=
295

296
### Tokenizer API
297

298
Our Tokenizer API is a simple wrapper over [HuggingFace-style tokenizers](https://huggingface.co/docs/transformers/en/main_classes/tokenizer).
299
300
301
302
303
It consists of two endpoints:

- `/tokenize` corresponds to calling `tokenizer.encode()`.
- `/detokenize` corresponds to calling `tokenizer.decode()`.

304
(pooling-api)=
305

306
307
308
309
310
311
### Pooling API

Our Pooling API encodes input prompts using a [pooling model](../models/pooling_models.md) and returns the corresponding hidden states.

The input format is the same as [Embeddings API](#embeddings-api), but the output data can contain an arbitrary nested list, not just a 1-D list of floats.

312
Code example: <gh-file:examples/online_serving/openai_pooling_client.py>
313

314
(score-api)=
315

316
317
### Score API

318
Our Score API applies a cross-encoder model to predict scores for sentence pairs.
319
320
321
322
Usually, the score for a sentence pair refers to the similarity between two sentences, on a scale of 0 to 1.

You can find the documentation for these kind of models at [sbert.net](https://www.sbert.net/docs/package_reference/cross_encoder/cross_encoder.html).

323
Code example: <gh-file:examples/online_serving/openai_cross_encoder_score.py>
324

325
326
327
328
329
#### Single inference

You can pass a string to both `text_1` and `text_2`, forming a single sentence pair.

Request:
330
331

```bash
332
333
334
335
336
337
338
339
340
341
curl -X 'POST' \
  'http://127.0.0.1:8000/score' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "BAAI/bge-reranker-v2-m3",
  "encoding_format": "float",
  "text_1": "What is the capital of France?",
  "text_2": "The capital of France is Paris."
}'
342
343
```

344
Response:
345

346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
```bash
{
  "id": "score-request-id",
  "object": "list",
  "created": 693447,
  "model": "BAAI/bge-reranker-v2-m3",
  "data": [
    {
      "index": 0,
      "object": "score",
      "score": 1
    }
  ],
  "usage": {}
}
361
362
```

363
#### Batch inference
364

365
366
367
You can pass a string to `text_1` and a list to `text_2`, forming multiple sentence pairs
where each pair is built from `text_1` and a string in `text_2`.
The total number of pairs is `len(text_2)`.
368

369
Request:
370

371
372
373
374
375
376
377
378
379
380
381
382
383
384
```bash
curl -X 'POST' \
  'http://127.0.0.1:8000/score' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "BAAI/bge-reranker-v2-m3",
  "text_1": "What is the capital of France?",
  "text_2": [
    "The capital of Brazil is Brasilia.",
    "The capital of France is Paris."
  ]
}'
```
385

386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
Response:

```bash
{
  "id": "score-request-id",
  "object": "list",
  "created": 693570,
  "model": "BAAI/bge-reranker-v2-m3",
  "data": [
    {
      "index": 0,
      "object": "score",
      "score": 0.001094818115234375
    },
    {
      "index": 1,
      "object": "score",
      "score": 1
    }
  ],
  "usage": {}
}
408
```
409

410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
You can pass a list to both `text_1` and `text_2`, forming multiple sentence pairs
where each pair is built from a string in `text_1` and the corresponding string in `text_2` (similar to `zip()`).
The total number of pairs is `len(text_2)`.

Request:

```bash
curl -X 'POST' \
  'http://127.0.0.1:8000/score' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "BAAI/bge-reranker-v2-m3",
  "encoding_format": "float",
  "text_1": [
    "What is the capital of Brazil?",
    "What is the capital of France?"
  ],
  "text_2": [
    "The capital of Brazil is Brasilia.",
    "The capital of France is Paris."
  ]
}'
```
434

435
Response:
436

437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
```bash
{
  "id": "score-request-id",
  "object": "list",
  "created": 693447,
  "model": "BAAI/bge-reranker-v2-m3",
  "data": [
    {
      "index": 0,
      "object": "score",
      "score": 1
    },
    {
      "index": 1,
      "object": "score",
      "score": 1
    }
  ],
  "usage": {}
}
```
458

459
#### Extra parameters
460

461
The following [pooling parameters](#pooling-params) are supported.
462

463
464
465
466
```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
:language: python
:start-after: begin-score-pooling-params
:end-before: end-score-pooling-params
467
468
```

469
470
471
472
473
474
The following extra parameters are supported:

```{literalinclude} ../../../vllm/entrypoints/openai/protocol.py
:language: python
:start-after: begin-score-extra-params
:end-before: end-score-extra-params
475
```