README.md 15.7 KB
Newer Older
Woosuk Kwon's avatar
Woosuk Kwon committed
1
# Benchmarking vLLM
2

3
4
5
This README guides you through running benchmark tests with the extensive
datasets supported on vLLM. It’s a living document, updated as new features and datasets
become available.
6

7
## Dataset Overview
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

<table style="width:100%; border-collapse: collapse;">
  <thead>
    <tr>
      <th style="width:15%; text-align: left;">Dataset</th>
      <th style="width:10%; text-align: center;">Online</th>
      <th style="width:10%; text-align: center;">Offline</th>
      <th style="width:65%; text-align: left;">Data Path</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>ShareGPT</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td><code>wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json</code></td>
    </tr>
    <tr>
      <td><strong>BurstGPT</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td><code>wget https://github.com/HPMLL/BurstGPT/releases/download/v1.1/BurstGPT_without_fails_2.csv</code></td>
    </tr>
    <tr>
      <td><strong>Sonnet</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td>Local file: <code>benchmarks/sonnet.txt</code></td>
    </tr>
    <tr>
      <td><strong>Random</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td><code>synthetic</code></td>
    </tr>
    <tr>
44
45
46
47
48
49
50
51
52
53
      <td><strong>HuggingFace-VisionArena</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td><code>lmarena-ai/VisionArena-Chat</code></td>
    </tr>
    <tr>
      <td><strong>HuggingFace-InstructCoder</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td><code>likaixin/InstructCoder</code></td>
54
55
56
57
58
59
    </tr>
      <tr>
      <td><strong>HuggingFace-AIMO</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td><code>AI-MO/aimo-validation-aime</code> , <code>AI-MO/NuminaMath-1.5</code>, <code>AI-MO/NuminaMath-CoT</code></td>
60
61
    </tr>
    <tr>
62
      <td><strong>HuggingFace-Other</strong></td>
63
      <td style="text-align: center;"></td>
64
      <td style="text-align: center;"></td>
65
      <td><code>lmms-lab/LLaVA-OneVision-Data</code>, <code>Aeala/ShareGPT_Vicuna_unfiltered</code></td>
66
    </tr>
67
68
69
70
71
72
    <tr>
      <td><strong>Custom</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td>Local file: <code>data.jsonl</code></td>
    </tr>
73
74
  </tbody>
</table>
75
76
77

✅: supported

78
🟡: Partial support
79

80
🚧: to be supported
81

82
**Note**: HuggingFace dataset's `dataset-name` should be set to `hf`
83

84
85
## 🚀 Example - Online Benchmark

86
<details>
87
<summary>Show more</summary>
88
89

<br/>
90
91

First start serving your model
92

93
```bash
94
vllm serve NousResearch/Hermes-3-Llama-3.1-8B --disable-log-requests
95
```
96

97
98
99
100
101
Then run the benchmarking script

```bash
# download dataset
# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
102
vllm bench serve \
103
104
105
106
107
108
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt \
  --dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 10
109
110
111
112
```

If successful, you will see the following output

113
```text
114
============ Serving Benchmark Result ============
115
116
117
118
119
120
121
Successful requests:                     10
Benchmark duration (s):                  5.78
Total input tokens:                      1369
Total generated tokens:                  2212
Request throughput (req/s):              1.73
Output token throughput (tok/s):         382.89
Total Token throughput (tok/s):          619.85
122
---------------Time to First Token----------------
123
124
125
Mean TTFT (ms):                          71.54
Median TTFT (ms):                        73.88
P99 TTFT (ms):                           79.49
126
-----Time per Output Token (excl. 1st token)------
127
128
129
Mean TPOT (ms):                          7.91
Median TPOT (ms):                        7.96
P99 TPOT (ms):                           8.03
130
---------------Inter-token Latency----------------
131
132
133
Mean ITL (ms):                           7.74
Median ITL (ms):                         7.70
P99 ITL (ms):                            8.39
134
135
==================================================
```
136

137
### Custom Dataset
138

139
140
If the dataset you want to benchmark is not supported yet in vLLM, even then you can benchmark on it using `CustomDataset`. Your data needs to be in `.jsonl` format and needs to have "prompt" field per entry, e.g., data.jsonl

141
```json
142
143
144
{"prompt": "What is the capital of India?"}
{"prompt": "What is the capital of Iran?"}
{"prompt": "What is the capital of China?"}
145
```
146
147
148
149
150
151
152
153

```bash
# start server
VLLM_USE_V1=1 vllm serve meta-llama/Llama-3.1-8B-Instruct --disable-log-requests
```

```bash
# run benchmarking script
154
vllm bench serve --port 9001 --save-result --save-detailed \
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
  --backend vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --endpoint /v1/completions \
  --dataset-name custom \
  --dataset-path <path-to-your-data-jsonl> \
  --custom-skip-chat-template \
  --num-prompts 80 \
  --max-concurrency 1 \
  --temperature=0.3 \
  --top-p=0.75 \
  --result-dir "./log/"
```

You can skip applying chat template if your data already has it by using `--custom-skip-chat-template`.

170
### VisionArena Benchmark for Vision Language Models
171

172
```bash
173
174
# need a model with vision capability here
vllm serve Qwen/Qwen2-VL-7B-Instruct --disable-log-requests
175
```
176

177
```bash
178
vllm bench serve \
179
180
181
182
183
184
185
  --backend openai-chat \
  --model Qwen/Qwen2-VL-7B-Instruct \
  --endpoint /v1/chat/completions \
  --dataset-name hf \
  --dataset-path lmarena-ai/VisionArena-Chat \
  --hf-split train \
  --num-prompts 1000
186
```
187

188
### InstructCoder Benchmark with Speculative Decoding
189

190
191
``` bash
VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
192
193
194
    --speculative-config $'{"method": "ngram",
    "num_speculative_tokens": 5, "prompt_lookup_max": 5,
    "prompt_lookup_min": 2}'
195
196
197
```

``` bash
198
vllm bench serve \
199
200
201
202
203
204
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --dataset-name hf \
    --dataset-path likaixin/InstructCoder \
    --num-prompts 2048
```

205
### Other HuggingFaceDataset Examples
206
207
208
209
210

```bash
vllm serve Qwen/Qwen2-VL-7B-Instruct --disable-log-requests
```

211
`lmms-lab/LLaVA-OneVision-Data`:
212
213

```bash
214
vllm bench serve \
215
216
217
218
219
220
221
222
  --backend openai-chat \
  --model Qwen/Qwen2-VL-7B-Instruct \
  --endpoint /v1/chat/completions \
  --dataset-name hf \
  --dataset-path lmms-lab/LLaVA-OneVision-Data \
  --hf-split train \
  --hf-subset "chart2text(cauldron)" \
  --num-prompts 10
223
224
```

225
`Aeala/ShareGPT_Vicuna_unfiltered`:
226
227

```bash
228
vllm bench serve \
229
230
231
232
233
234
235
  --backend openai-chat \
  --model Qwen/Qwen2-VL-7B-Instruct \
  --endpoint /v1/chat/completions \
  --dataset-name hf \
  --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \
  --hf-split train \
  --num-prompts 10
236
237
```

238
`AI-MO/aimo-validation-aime`:
239
240

``` bash
241
vllm bench serve \
242
243
244
245
246
247
248
    --model Qwen/QwQ-32B \
    --dataset-name hf \
    --dataset-path AI-MO/aimo-validation-aime \
    --num-prompts 10 \
    --seed 42
```

249
`philschmid/mt-bench`:
250
251

``` bash
252
vllm bench serve \
253
254
255
256
257
258
    --model Qwen/QwQ-32B \
    --dataset-name hf \
    --dataset-path philschmid/mt-bench \
    --num-prompts 80
```

259
### Running With Sampling Parameters
260
261
262
263
264

When using OpenAI-compatible backends such as `vllm`, optional sampling
parameters can be specified. Example client command:

```bash
265
vllm bench serve \
266
267
268
269
270
271
272
273
274
275
276
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt \
  --dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
  --top-k 10 \
  --top-p 0.9 \
  --temperature 0.5 \
  --num-prompts 10
```

277
### Running With Ramp-Up Request Rate
278
279
280
281
282
283

The benchmark tool also supports ramping up the request rate over the
duration of the benchmark run. This can be useful for stress testing the
server or finding the maximum throughput that it can handle, given some latency budget.

Two ramp-up strategies are supported:
284

285
286
287
288
- `linear`: Increases the request rate linearly from a start value to an end value.
- `exponential`: Increases the request rate exponentially.

The following arguments can be used to control the ramp-up:
289

290
291
292
293
- `--ramp-up-strategy`: The ramp-up strategy to use (`linear` or `exponential`).
- `--ramp-up-start-rps`: The request rate at the beginning of the benchmark.
- `--ramp-up-end-rps`: The request rate at the end of the benchmark.

294
295
</details>

296
297
## 📈 Example - Offline Throughput Benchmark

298
<details>
299
<summary>Show more</summary>
300
301

<br/>
302
303

```bash
304
vllm bench throughput \
305
306
307
308
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --dataset-name sonnet \
  --dataset-path vllm/benchmarks/sonnet.txt \
  --num-prompts 10
309
```
310
311
312

If successful, you will see the following output

313
```text
314
315
316
317
318
Throughput: 7.15 requests/s, 4656.00 total tokens/s, 1072.15 output tokens/s
Total num prompt tokens:  5014
Total num output tokens:  1500
```

319
### VisionArena Benchmark for Vision Language Models
320

321
```bash
322
vllm bench throughput \
323
324
325
326
327
328
  --model Qwen/Qwen2-VL-7B-Instruct \
  --backend vllm-chat \
  --dataset-name hf \
  --dataset-path lmarena-ai/VisionArena-Chat \
  --num-prompts 1000 \
  --hf-split train
329
330
331
332
```

The `num prompt tokens` now includes image token counts

333
```text
334
335
336
Throughput: 2.55 requests/s, 4036.92 total tokens/s, 326.90 output tokens/s
Total num prompt tokens:  14527
Total num output tokens:  1280
337
```
338

339
### InstructCoder Benchmark with Speculative Decoding
340
341
342
343

``` bash
VLLM_WORKER_MULTIPROC_METHOD=spawn \
VLLM_USE_V1=1 \
344
vllm bench throughput \
345
346
347
348
349
350
351
    --dataset-name=hf \
    --dataset-path=likaixin/InstructCoder \
    --model=meta-llama/Meta-Llama-3-8B-Instruct \
    --input-len=1000 \
    --output-len=100 \
    --num-prompts=2048 \
    --async-engine \
352
353
354
    --speculative-config $'{"method": "ngram",
    "num_speculative_tokens": 5, "prompt_lookup_max": 5,
    "prompt_lookup_min": 2}'
355
356
```

357
```text
358
359
360
361
362
Throughput: 104.77 requests/s, 23836.22 total tokens/s, 10477.10 output tokens/s
Total num prompt tokens:  261136
Total num output tokens:  204800
```

363
### Other HuggingFaceDataset Examples
364

365
`lmms-lab/LLaVA-OneVision-Data`:
366
367

```bash
368
vllm bench throughput \
369
370
371
372
373
374
375
376
377
  --model Qwen/Qwen2-VL-7B-Instruct \
  --backend vllm-chat \
  --dataset-name hf \
  --dataset-path lmms-lab/LLaVA-OneVision-Data \
  --hf-split train \
  --hf-subset "chart2text(cauldron)" \
  --num-prompts 10
```

378
`Aeala/ShareGPT_Vicuna_unfiltered`:
379
380

```bash
381
vllm bench throughput \
382
383
384
385
386
387
388
389
  --model Qwen/Qwen2-VL-7B-Instruct \
  --backend vllm-chat \
  --dataset-name hf \
  --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \
  --hf-split train \
  --num-prompts 10
```

390
`AI-MO/aimo-validation-aime`:
391
392

```bash
393
vllm bench throughput \
394
395
396
397
398
399
400
401
  --model Qwen/QwQ-32B \
  --backend vllm \
  --dataset-name hf \
  --dataset-path AI-MO/aimo-validation-aime \
  --hf-split train \
  --num-prompts 10
```

402
Benchmark with LoRA adapters:
403
404

``` bash
405
406
# download dataset
# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
407
vllm bench throughput \
408
409
410
411
412
413
414
415
416
  --model meta-llama/Llama-2-7b-hf \
  --backend vllm \
  --dataset_path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
  --dataset_name sharegpt \
  --num-prompts 10 \
  --max-loras 2 \
  --max-lora-rank 8 \
  --enable-lora \
  --lora-path yard1/llama-2-7b-sql-lora-test
417
  ```
418

419
420
</details>

421
422
## 🛠️ Example - Structured Output Benchmark

423
<details>
424
<summary>Show more</summary>
425
426

<br/>
427
428
429

Benchmark the performance of structured output generation (JSON, grammar, regex).

430
### Server Setup
431
432
433
434
435

```bash
vllm serve NousResearch/Hermes-3-Llama-3.1-8B --disable-log-requests
```

436
### JSON Schema Benchmark
437
438
439
440
441
442
443
444
445
446
447

```bash
python3 benchmarks/benchmark_serving_structured_output.py \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --dataset json \
  --structured-output-ratio 1.0 \
  --request-rate 10 \
  --num-prompts 1000
```

448
### Grammar-based Generation Benchmark
449
450
451
452
453
454
455
456
457
458
459

```bash
python3 benchmarks/benchmark_serving_structured_output.py \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --dataset grammar \
  --structure-type grammar \
  --request-rate 10 \
  --num-prompts 1000
```

460
### Regex-based Generation Benchmark
461
462
463
464
465
466
467
468
469
470

```bash
python3 benchmarks/benchmark_serving_structured_output.py \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --dataset regex \
  --request-rate 10 \
  --num-prompts 1000
```

471
### Choice-based Generation Benchmark
472
473
474
475
476
477
478
479
480
481

```bash
python3 benchmarks/benchmark_serving_structured_output.py \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --dataset choice \
  --request-rate 10 \
  --num-prompts 1000
```

482
### XGrammar Benchmark Dataset
483
484
485
486
487
488
489
490
491
492

```bash
python3 benchmarks/benchmark_serving_structured_output.py \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --dataset xgrammar_bench \
  --request-rate 10 \
  --num-prompts 1000
```

493
494
</details>

495
496
## 📚 Example - Long Document QA Benchmark

497
<details>
498
<summary>Show more</summary>
499
500

<br/>
501
502
503

Benchmark the performance of long document question-answering with prefix caching.

504
### Basic Long Document QA Test
505
506
507
508
509
510
511
512
513
514
515

```bash
python3 benchmarks/benchmark_long_document_qa_throughput.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --enable-prefix-caching \
  --num-documents 16 \
  --document-length 2000 \
  --output-len 50 \
  --repeat-count 5
```

516
### Different Repeat Modes
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546

```bash
# Random mode (default) - shuffle prompts randomly
python3 benchmarks/benchmark_long_document_qa_throughput.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --enable-prefix-caching \
  --num-documents 8 \
  --document-length 3000 \
  --repeat-count 3 \
  --repeat-mode random

# Tile mode - repeat entire prompt list in sequence
python3 benchmarks/benchmark_long_document_qa_throughput.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --enable-prefix-caching \
  --num-documents 8 \
  --document-length 3000 \
  --repeat-count 3 \
  --repeat-mode tile

# Interleave mode - repeat each prompt consecutively
python3 benchmarks/benchmark_long_document_qa_throughput.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --enable-prefix-caching \
  --num-documents 8 \
  --document-length 3000 \
  --repeat-count 3 \
  --repeat-mode interleave
```

547
548
</details>

549
550
## 🗂️ Example - Prefix Caching Benchmark

551
<details>
552
<summary>Show more</summary>
553
554

<br/>
555
556
557

Benchmark the efficiency of automatic prefix caching.

558
### Fixed Prompt with Prefix Caching
559
560
561
562
563
564
565
566
567
568

```bash
python3 benchmarks/benchmark_prefix_caching.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --enable-prefix-caching \
  --num-prompts 1 \
  --repeat-count 100 \
  --input-length-range 128:256
```

569
### ShareGPT Dataset with Prefix Caching
570
571
572
573
574
575
576
577
578
579
580
581
582
583

```bash
# download dataset
# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

python3 benchmarks/benchmark_prefix_caching.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --dataset-path /path/ShareGPT_V3_unfiltered_cleaned_split.json \
  --enable-prefix-caching \
  --num-prompts 20 \
  --repeat-count 5 \
  --input-length-range 128:256
```

584
585
</details>

586
587
## ⚡ Example - Request Prioritization Benchmark

588
<details>
589
<summary>Show more</summary>
590
591

<br/>
592
593
594

Benchmark the performance of request prioritization in vLLM.

595
### Basic Prioritization Test
596
597
598
599
600
601
602
603
604
605

```bash
python3 benchmarks/benchmark_prioritization.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --input-len 128 \
  --output-len 64 \
  --num-prompts 100 \
  --scheduling-policy priority
```

606
### Multiple Sequences per Prompt
607
608
609
610
611
612
613
614
615
616

```bash
python3 benchmarks/benchmark_prioritization.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --input-len 128 \
  --output-len 64 \
  --num-prompts 100 \
  --scheduling-policy priority \
  --n 2
```
617
618

</details>