README.md 17.6 KB
Newer Older
Woosuk Kwon's avatar
Woosuk Kwon committed
1
# Benchmarking vLLM
2

3
4
5
This README guides you through running benchmark tests with the extensive
datasets supported on vLLM. It’s a living document, updated as new features and datasets
become available.
6

7
## Dataset Overview
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

<table style="width:100%; border-collapse: collapse;">
  <thead>
    <tr>
      <th style="width:15%; text-align: left;">Dataset</th>
      <th style="width:10%; text-align: center;">Online</th>
      <th style="width:10%; text-align: center;">Offline</th>
      <th style="width:65%; text-align: left;">Data Path</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>ShareGPT</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td><code>wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json</code></td>
    </tr>
25
26
27
28
29
30
31
32
33
34
35
    <tr>
      <td><strong>ShareGPT4V (Image)</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td>
        <code>wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/blob/main/sharegpt4v_instruct_gpt4-vision_cap100k.json</code>
        <br>
        <div>Note that the images need to be downloaded separately. For example, to download COCO's 2017 Train images:</div>
        <code>wget http://images.cocodataset.org/zips/train2017.zip</code>
      </td>
    </tr>
36
37
38
39
40
41
42
    <tr>
      <td><strong>BurstGPT</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td><code>wget https://github.com/HPMLL/BurstGPT/releases/download/v1.1/BurstGPT_without_fails_2.csv</code></td>
    </tr>
    <tr>
43
      <td><strong>Sonnet (deprecated)</strong></td>
44
45
46
47
48
49
50
51
52
53
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td>Local file: <code>benchmarks/sonnet.txt</code></td>
    </tr>
    <tr>
      <td><strong>Random</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td><code>synthetic</code></td>
    </tr>
54
55
56
57
58
59
    <tr>
      <td><strong>Prefix Repetition</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td><code>synthetic</code></td>
    </tr>
60
    <tr>
61
62
63
64
65
66
67
68
69
70
      <td><strong>HuggingFace-VisionArena</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td><code>lmarena-ai/VisionArena-Chat</code></td>
    </tr>
    <tr>
      <td><strong>HuggingFace-InstructCoder</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td><code>likaixin/InstructCoder</code></td>
71
72
73
74
75
76
    </tr>
      <tr>
      <td><strong>HuggingFace-AIMO</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td><code>AI-MO/aimo-validation-aime</code> , <code>AI-MO/NuminaMath-1.5</code>, <code>AI-MO/NuminaMath-CoT</code></td>
77
78
    </tr>
    <tr>
79
      <td><strong>HuggingFace-Other</strong></td>
80
      <td style="text-align: center;"></td>
81
      <td style="text-align: center;"></td>
82
      <td><code>lmms-lab/LLaVA-OneVision-Data</code>, <code>Aeala/ShareGPT_Vicuna_unfiltered</code></td>
83
    </tr>
84
85
86
87
88
89
    <tr>
      <td><strong>Custom</strong></td>
      <td style="text-align: center;"></td>
      <td style="text-align: center;"></td>
      <td>Local file: <code>data.jsonl</code></td>
    </tr>
90
91
  </tbody>
</table>
92
93
94

✅: supported

95
🟡: Partial support
96

97
🚧: to be supported
98

99
**Note**: HuggingFace dataset's `dataset-name` should be set to `hf`
100

101
102
## 🚀 Example - Online Benchmark

103
<details>
104
<summary>Show more</summary>
105
106

<br/>
107
108

First start serving your model
109

110
```bash
111
vllm serve NousResearch/Hermes-3-Llama-3.1-8B
112
```
113

114
115
116
117
118
Then run the benchmarking script

```bash
# download dataset
# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
119
vllm bench serve \
120
121
122
123
124
125
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt \
  --dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 10
126
127
128
129
```

If successful, you will see the following output

130
```text
131
============ Serving Benchmark Result ============
132
133
134
135
136
137
138
Successful requests:                     10
Benchmark duration (s):                  5.78
Total input tokens:                      1369
Total generated tokens:                  2212
Request throughput (req/s):              1.73
Output token throughput (tok/s):         382.89
Total Token throughput (tok/s):          619.85
139
---------------Time to First Token----------------
140
141
142
Mean TTFT (ms):                          71.54
Median TTFT (ms):                        73.88
P99 TTFT (ms):                           79.49
143
-----Time per Output Token (excl. 1st token)------
144
145
146
Mean TPOT (ms):                          7.91
Median TPOT (ms):                        7.96
P99 TPOT (ms):                           8.03
147
---------------Inter-token Latency----------------
148
149
150
Mean ITL (ms):                           7.74
Median ITL (ms):                         7.70
P99 ITL (ms):                            8.39
151
152
==================================================
```
153

154
### Custom Dataset
155

156
157
If the dataset you want to benchmark is not supported yet in vLLM, even then you can benchmark on it using `CustomDataset`. Your data needs to be in `.jsonl` format and needs to have "prompt" field per entry, e.g., data.jsonl

158
```json
159
160
161
{"prompt": "What is the capital of India?"}
{"prompt": "What is the capital of Iran?"}
{"prompt": "What is the capital of China?"}
162
```
163
164
165

```bash
# start server
166
VLLM_USE_V1=1 vllm serve meta-llama/Llama-3.1-8B-Instruct
167
168
169
170
```

```bash
# run benchmarking script
171
vllm bench serve --port 9001 --save-result --save-detailed \
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
  --backend vllm \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --endpoint /v1/completions \
  --dataset-name custom \
  --dataset-path <path-to-your-data-jsonl> \
  --custom-skip-chat-template \
  --num-prompts 80 \
  --max-concurrency 1 \
  --temperature=0.3 \
  --top-p=0.75 \
  --result-dir "./log/"
```

You can skip applying chat template if your data already has it by using `--custom-skip-chat-template`.

187
### VisionArena Benchmark for Vision Language Models
188

189
```bash
190
# need a model with vision capability here
191
vllm serve Qwen/Qwen2-VL-7B-Instruct
192
```
193

194
```bash
195
vllm bench serve \
196
  --backend openai-chat \
197
  --endpoint-type openai-chat \
198
199
200
201
202
203
  --model Qwen/Qwen2-VL-7B-Instruct \
  --endpoint /v1/chat/completions \
  --dataset-name hf \
  --dataset-path lmarena-ai/VisionArena-Chat \
  --hf-split train \
  --num-prompts 1000
204
```
205

206
### InstructCoder Benchmark with Speculative Decoding
207

208
209
``` bash
VLLM_USE_V1=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
210
211
212
    --speculative-config $'{"method": "ngram",
    "num_speculative_tokens": 5, "prompt_lookup_max": 5,
    "prompt_lookup_min": 2}'
213
214
215
```

``` bash
216
vllm bench serve \
217
218
219
220
221
222
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --dataset-name hf \
    --dataset-path likaixin/InstructCoder \
    --num-prompts 2048
```

223
### Other HuggingFaceDataset Examples
224
225

```bash
226
vllm serve Qwen/Qwen2-VL-7B-Instruct
227
228
```

229
`lmms-lab/LLaVA-OneVision-Data`:
230
231

```bash
232
vllm bench serve \
233
  --backend openai-chat \
234
  --endpoint-type openai-chat \  
235
236
237
238
239
240
241
  --model Qwen/Qwen2-VL-7B-Instruct \
  --endpoint /v1/chat/completions \
  --dataset-name hf \
  --dataset-path lmms-lab/LLaVA-OneVision-Data \
  --hf-split train \
  --hf-subset "chart2text(cauldron)" \
  --num-prompts 10
242
243
```

244
`Aeala/ShareGPT_Vicuna_unfiltered`:
245
246

```bash
247
vllm bench serve \
248
  --backend openai-chat \
249
  --endpoint-type openai-chat \  
250
251
252
253
254
255
  --model Qwen/Qwen2-VL-7B-Instruct \
  --endpoint /v1/chat/completions \
  --dataset-name hf \
  --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \
  --hf-split train \
  --num-prompts 10
256
257
```

258
`AI-MO/aimo-validation-aime`:
259
260

``` bash
261
vllm bench serve \
262
263
264
265
266
267
268
    --model Qwen/QwQ-32B \
    --dataset-name hf \
    --dataset-path AI-MO/aimo-validation-aime \
    --num-prompts 10 \
    --seed 42
```

269
`philschmid/mt-bench`:
270
271

``` bash
272
vllm bench serve \
273
274
275
276
277
278
    --model Qwen/QwQ-32B \
    --dataset-name hf \
    --dataset-path philschmid/mt-bench \
    --num-prompts 80
```

279
### Running With Sampling Parameters
280
281
282
283
284

When using OpenAI-compatible backends such as `vllm`, optional sampling
parameters can be specified. Example client command:

```bash
285
vllm bench serve \
286
287
288
289
290
291
292
293
294
295
296
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --endpoint /v1/completions \
  --dataset-name sharegpt \
  --dataset-path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
  --top-k 10 \
  --top-p 0.9 \
  --temperature 0.5 \
  --num-prompts 10
```

297
### Running With Ramp-Up Request Rate
298
299
300
301
302
303

The benchmark tool also supports ramping up the request rate over the
duration of the benchmark run. This can be useful for stress testing the
server or finding the maximum throughput that it can handle, given some latency budget.

Two ramp-up strategies are supported:
304

305
306
307
308
- `linear`: Increases the request rate linearly from a start value to an end value.
- `exponential`: Increases the request rate exponentially.

The following arguments can be used to control the ramp-up:
309

310
311
312
313
- `--ramp-up-strategy`: The ramp-up strategy to use (`linear` or `exponential`).
- `--ramp-up-start-rps`: The request rate at the beginning of the benchmark.
- `--ramp-up-end-rps`: The request rate at the end of the benchmark.

314
315
</details>

316
317
## 📈 Example - Offline Throughput Benchmark

318
<details>
319
<summary>Show more</summary>
320
321

<br/>
322
323

```bash
324
vllm bench throughput \
325
326
327
328
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --dataset-name sonnet \
  --dataset-path vllm/benchmarks/sonnet.txt \
  --num-prompts 10
329
```
330
331
332

If successful, you will see the following output

333
```text
334
335
336
337
338
Throughput: 7.15 requests/s, 4656.00 total tokens/s, 1072.15 output tokens/s
Total num prompt tokens:  5014
Total num output tokens:  1500
```

339
### VisionArena Benchmark for Vision Language Models
340

341
```bash
342
vllm bench throughput \
343
344
345
346
347
348
  --model Qwen/Qwen2-VL-7B-Instruct \
  --backend vllm-chat \
  --dataset-name hf \
  --dataset-path lmarena-ai/VisionArena-Chat \
  --num-prompts 1000 \
  --hf-split train
349
350
351
352
```

The `num prompt tokens` now includes image token counts

353
```text
354
355
356
Throughput: 2.55 requests/s, 4036.92 total tokens/s, 326.90 output tokens/s
Total num prompt tokens:  14527
Total num output tokens:  1280
357
```
358

359
### InstructCoder Benchmark with Speculative Decoding
360
361
362
363

``` bash
VLLM_WORKER_MULTIPROC_METHOD=spawn \
VLLM_USE_V1=1 \
364
vllm bench throughput \
365
366
367
368
369
370
371
    --dataset-name=hf \
    --dataset-path=likaixin/InstructCoder \
    --model=meta-llama/Meta-Llama-3-8B-Instruct \
    --input-len=1000 \
    --output-len=100 \
    --num-prompts=2048 \
    --async-engine \
372
373
374
    --speculative-config $'{"method": "ngram",
    "num_speculative_tokens": 5, "prompt_lookup_max": 5,
    "prompt_lookup_min": 2}'
375
376
```

377
```text
378
379
380
381
382
Throughput: 104.77 requests/s, 23836.22 total tokens/s, 10477.10 output tokens/s
Total num prompt tokens:  261136
Total num output tokens:  204800
```

383
### Other HuggingFaceDataset Examples
384

385
`lmms-lab/LLaVA-OneVision-Data`:
386
387

```bash
388
vllm bench throughput \
389
390
391
392
393
394
395
396
397
  --model Qwen/Qwen2-VL-7B-Instruct \
  --backend vllm-chat \
  --dataset-name hf \
  --dataset-path lmms-lab/LLaVA-OneVision-Data \
  --hf-split train \
  --hf-subset "chart2text(cauldron)" \
  --num-prompts 10
```

398
`Aeala/ShareGPT_Vicuna_unfiltered`:
399
400

```bash
401
vllm bench throughput \
402
403
404
405
406
407
408
409
  --model Qwen/Qwen2-VL-7B-Instruct \
  --backend vllm-chat \
  --dataset-name hf \
  --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \
  --hf-split train \
  --num-prompts 10
```

410
`AI-MO/aimo-validation-aime`:
411
412

```bash
413
vllm bench throughput \
414
415
416
417
418
419
420
421
  --model Qwen/QwQ-32B \
  --backend vllm \
  --dataset-name hf \
  --dataset-path AI-MO/aimo-validation-aime \
  --hf-split train \
  --num-prompts 10
```

422
Benchmark with LoRA adapters:
423
424

``` bash
425
426
# download dataset
# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
427
vllm bench throughput \
428
429
430
431
432
433
434
435
436
  --model meta-llama/Llama-2-7b-hf \
  --backend vllm \
  --dataset_path <your data path>/ShareGPT_V3_unfiltered_cleaned_split.json \
  --dataset_name sharegpt \
  --num-prompts 10 \
  --max-loras 2 \
  --max-lora-rank 8 \
  --enable-lora \
  --lora-path yard1/llama-2-7b-sql-lora-test
437
  ```
438

439
440
</details>

441
442
## 🛠️ Example - Structured Output Benchmark

443
<details>
444
<summary>Show more</summary>
445
446

<br/>
447
448
449

Benchmark the performance of structured output generation (JSON, grammar, regex).

450
### Server Setup
451
452

```bash
453
vllm serve NousResearch/Hermes-3-Llama-3.1-8B
454
455
```

456
### JSON Schema Benchmark
457
458
459
460
461
462
463
464
465
466
467

```bash
python3 benchmarks/benchmark_serving_structured_output.py \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --dataset json \
  --structured-output-ratio 1.0 \
  --request-rate 10 \
  --num-prompts 1000
```

468
### Grammar-based Generation Benchmark
469
470
471
472
473
474
475
476
477
478
479

```bash
python3 benchmarks/benchmark_serving_structured_output.py \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --dataset grammar \
  --structure-type grammar \
  --request-rate 10 \
  --num-prompts 1000
```

480
### Regex-based Generation Benchmark
481
482
483
484
485
486
487
488
489
490

```bash
python3 benchmarks/benchmark_serving_structured_output.py \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --dataset regex \
  --request-rate 10 \
  --num-prompts 1000
```

491
### Choice-based Generation Benchmark
492
493
494
495
496
497
498
499
500
501

```bash
python3 benchmarks/benchmark_serving_structured_output.py \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --dataset choice \
  --request-rate 10 \
  --num-prompts 1000
```

502
### XGrammar Benchmark Dataset
503
504
505
506
507
508
509
510
511
512

```bash
python3 benchmarks/benchmark_serving_structured_output.py \
  --backend vllm \
  --model NousResearch/Hermes-3-Llama-3.1-8B \
  --dataset xgrammar_bench \
  --request-rate 10 \
  --num-prompts 1000
```

513
514
</details>

515
516
## 📚 Example - Long Document QA Benchmark

517
<details>
518
<summary>Show more</summary>
519
520

<br/>
521
522
523

Benchmark the performance of long document question-answering with prefix caching.

524
### Basic Long Document QA Test
525
526
527
528
529
530
531
532
533
534
535

```bash
python3 benchmarks/benchmark_long_document_qa_throughput.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --enable-prefix-caching \
  --num-documents 16 \
  --document-length 2000 \
  --output-len 50 \
  --repeat-count 5
```

536
### Different Repeat Modes
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566

```bash
# Random mode (default) - shuffle prompts randomly
python3 benchmarks/benchmark_long_document_qa_throughput.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --enable-prefix-caching \
  --num-documents 8 \
  --document-length 3000 \
  --repeat-count 3 \
  --repeat-mode random

# Tile mode - repeat entire prompt list in sequence
python3 benchmarks/benchmark_long_document_qa_throughput.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --enable-prefix-caching \
  --num-documents 8 \
  --document-length 3000 \
  --repeat-count 3 \
  --repeat-mode tile

# Interleave mode - repeat each prompt consecutively
python3 benchmarks/benchmark_long_document_qa_throughput.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --enable-prefix-caching \
  --num-documents 8 \
  --document-length 3000 \
  --repeat-count 3 \
  --repeat-mode interleave
```

567
568
</details>

569
570
## 🗂️ Example - Prefix Caching Benchmark

571
<details>
572
<summary>Show more</summary>
573
574

<br/>
575
576
577

Benchmark the efficiency of automatic prefix caching.

578
### Fixed Prompt with Prefix Caching
579
580
581
582
583
584
585
586
587
588

```bash
python3 benchmarks/benchmark_prefix_caching.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --enable-prefix-caching \
  --num-prompts 1 \
  --repeat-count 100 \
  --input-length-range 128:256
```

589
### ShareGPT Dataset with Prefix Caching
590
591
592
593
594
595
596
597
598
599
600
601
602
603

```bash
# download dataset
# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

python3 benchmarks/benchmark_prefix_caching.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --dataset-path /path/ShareGPT_V3_unfiltered_cleaned_split.json \
  --enable-prefix-caching \
  --num-prompts 20 \
  --repeat-count 5 \
  --input-length-range 128:256
```

604
605
606
607
608
609
610
611
612
613
614
615
616
617
### Prefix Repetition Dataset

```bash
vllm bench serve \
  --backend openai \
  --model meta-llama/Llama-2-7b-chat-hf \
  --dataset-name prefix_repetition \
  --num-prompts 100 \
  --prefix-repetition-prefix-len 512 \
  --prefix-repetition-suffix-len 128 \
  --prefix-repetition-num-prefixes 5 \
  --prefix-repetition-output-len 128 
```

618
619
</details>

620
621
## ⚡ Example - Request Prioritization Benchmark

622
<details>
623
<summary>Show more</summary>
624
625

<br/>
626
627
628

Benchmark the performance of request prioritization in vLLM.

629
### Basic Prioritization Test
630
631
632
633
634
635
636
637
638
639

```bash
python3 benchmarks/benchmark_prioritization.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --input-len 128 \
  --output-len 64 \
  --num-prompts 100 \
  --scheduling-policy priority
```

640
### Multiple Sequences per Prompt
641
642
643
644
645
646
647
648
649
650

```bash
python3 benchmarks/benchmark_prioritization.py \
  --model meta-llama/Llama-2-7b-chat-hf \
  --input-len 128 \
  --output-len 64 \
  --num-prompts 100 \
  --scheduling-policy priority \
  --n 2
```
651
652

</details>
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690

## 👁️ Example - Multi-Modal Benchmark

<details>
<summary>Show more</summary>

<br/>

Benchmark the performance of multi-modal requests in vLLM.

### Images (ShareGPT4V)

Start vLLM:

```bash
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --dtype bfloat16 \
  --limit-mm-per-prompt '{"image": 1}' \
  --allowed-local-media-path /path/to/sharegpt4v/images
```

Send requests with images:

```bash
python benchmarks/benchmark_serving.py \
  --backend openai-chat \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --dataset-name sharegpt \
  --dataset-path /path/to/ShareGPT4V/sharegpt4v_instruct_gpt4-vision_cap100k.json \
  --num-prompts 100 \
  --save-result \
  --result-dir ~/vllm_benchmark_results \
  --save-detailed \
  --endpoint /v1/chat/completion
```

</details>