Commits · b6ae5aeca66f8222cb2eec392f3b5330dc340484 · OpenDAS / vllm_cscc

02 Dec, 2025 2 commits
- [Bugfix][EPLB] Prevent user-provided EPLB config from being overwritten with defaults (#29911) · b6ae5aec
  Sage Moore authored Dec 02, 2025
```
Signed-off-by: Sage Moore <sage@neuralmagic.com>
(cherry picked from commit e6f114ac)
```
  b6ae5aec
- [Core] Support reseting all running requests' KV while calling `reset_prefix_cache` (#28827) · d0cd7289
  Zhuohan Li authored Dec 01, 2025
```
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
```
  d0cd7289
01 Dec, 2025 2 commits

[Core][Observability] Add KV cache residency metrics (#27793) · cabc77cc

shivampr authored Dec 01, 2025



Introduces three new Prometheus histograms for fine-grained observability of KV cache residency behavior:

vllm:kv_block_lifetime_seconds — total lifetime from allocation to free
vllm:kv_block_idle_before_evict_seconds — idle duration before eviction
vllm:kv_block_reuse_gap_seconds — time between consecutive reuses of the same block

These metrics help operators analyze KV cache efficiency, reuse patterns, and eviction timing beyond simple utilization rates.

Implementation uses monotonic timestamps for accuracy, 1% sampling for minimal overhead (~48 bytes/block), and is fully thread-safe with zero runtime cost when disabled.

Two new runtime flags are introduced:

--kv-cache-metrics – enable KV cache residency metrics
--kv-cache-metrics-sample – control sampling ratio (default: 0.01)
Signed-off-by: Shivam <shivamprasad91@gmail.com>

cabc77cc

[Misc] Unify tokenizer registration (#29767) · f0a28bf6
Cyrus Leung authored Dec 01, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
f0a28bf6

29 Nov, 2025 2 commits
- [Misc] Refactor tokenizer interface (#29693) · 34a98427
  Cyrus Leung authored Nov 29, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  34a98427
- [Frontend] Perform offline path replacement to `tokenizer` (#29706) · 762a4a6c
  Tsukasa OI authored Nov 29, 2025
```
Signed-off-by: Tsukasa OI <floss_llm@irq.a4lg.com>
```
  762a4a6c
28 Nov, 2025 5 commits
- [Frontend] Remap -O to -cc commandline flag (#29557) · 3461e7ef
  Yanan Cao authored Nov 28, 2025
```
Signed-off-by: Yanan Cao <gmagogsfm@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
```
  3461e7ef
- [Chore] Rename `Processor` to `InputProcessor` (#29682) · 8d9338fa
  Cyrus Leung authored Nov 29, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  8d9338fa
- [Chore]: Reorganize model repo operating functions in `transformers_utils` (#29680) · f946a8d7
  Isotr0py authored Nov 29, 2025
```
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
```
  f946a8d7
- [mypy] Enable type checking for more directories (#29674) · 9e6bcda3
  Cyrus Leung authored Nov 29, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  9e6bcda3
- Improve enable chunked_prefill & prefix_caching logic. (#26623) · f4b76056
  wang.yuqi authored Nov 28, 2025
```
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
```
  f4b76056
27 Nov, 2025 2 commits

[Misc] Remove unused code from `protocol.py` (#29616) · ea228b44
Cyrus Leung authored Nov 28, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
ea228b44

[Frontend][torch.compile] CompilationConfig Overhaul (#20283): Set up -O infrastructure (#26847) · 0838b52e

Morrison Turnansky authored Nov 27, 2025


Signed-off-by: morrison-turnansky <mturnans@redhat.com>
Signed-off-by: adabeyta <aabeyta@redhat.com>
Signed-off-by: Morrison Turnansky <mturnans@redhat.com>
Co-authored-by: adabeyta <aabeyta@redhat.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

0838b52e

25 Nov, 2025 8 commits

Scheduled removal of `override_pooler_config` and `disable_log_requests` (#29402) · a1f26768

Harry Mellor authored Nov 25, 2025


Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

a1f26768

[Hybrid Allocator] Support KV cache groups with different block_size (#29143) · 48ddb02b
Yifan Qiao authored Nov 25, 2025
```
Signed-off-by: Yifan Qiao <yifanqiao@berkeley.edu>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
```
48ddb02b

[Feature]: Improve GGUF loading from HuggingFace user experience like repo_id:quant_type (#29137) · 794029f0

Injae Ryou authored Nov 25, 2025


Signed-off-by: Injae Ryou <injaeryou@gmail.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>

794029f0

[Bugfix] Fix logic for choosing default prefix caching setting (#29393) · 516c3f78
Thomas Parnell authored Nov 25, 2025
```
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
```
516c3f78

[Misc] Suppress log outputs when constructing the default vllm config. (#29291) · de688994

wang.yuqi authored Nov 25, 2025

Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Signed-off-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

de688994

[Perf] use cpu all reduce to avoid sync when async_scheduling & dp > 1 (#29311) · f242cfcd
zhrrr authored Nov 25, 2025
```
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
```
f242cfcd
Scheduled removal of `guided_*` config fields (#29326) · 316c8492
Harry Mellor authored Nov 25, 2025
```
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
```
316c8492
Scheduled removal of `ParallelConfig`'s direct child EPLB fields (#29324) · a4ad43ad
Harry Mellor authored Nov 25, 2025
```
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
```
a4ad43ad

22 Nov, 2025 2 commits
- [perf][cpu] Accelerate paged attention GEMMs (QK, PV) on Arm CPUs with NEON (#29193) · 730bd353
  Fadi Arafeh authored Nov 22, 2025
```
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>
```
  730bd353
- [Misc] Further clean up chunked prefill and prefix caching init (#29186) · 5a480258
  Cyrus Leung authored Nov 22, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  5a480258
21 Nov, 2025 3 commits
- [Deprecation] Deprecate `seed=None` (#29185) · ceca0605
  Cyrus Leung authored Nov 22, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  ceca0605
- [Misc] Move dynamic seed initialization to `EngineArgs` (#29165) · d7219bcd
  Cyrus Leung authored Nov 21, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  d7219bcd
- [LoRA][2/2]Remove LoRA extra vocab (#28545) · 9875be64
  Jee Jee Li authored Nov 21, 2025
```
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
```
  9875be64
20 Nov, 2025 2 commits

[RL] Add Pause and Resume Generation for Asynchronous RL Training (#28037) · 371b1d4c

Samit authored Nov 20, 2025


Signed-off-by: SamitHuang <285365963@qq.com>
Signed-off-by: Samit <285365963@qq.com>
Signed-off-by: samithuang <285365963@qq.com>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>

371b1d4c

[V0 Deprecation] Remove `num_lookahead_slots` (#29000) · 20e4497b

Cyrus Leung authored Nov 20, 2025


Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Michael Goin <mgoin64@gmail.com>

20e4497b

19 Nov, 2025 2 commits

[Feature] Prefill Context Parallel (PCP) basic support (#28718) · 2fd893b4

Qiu authored Nov 20, 2025


Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Signed-off-by: FENP <yuanyongjie.yyj@antgroup.com>
Signed-off-by: LookAround <lixushi@huawei.com>
Signed-off-by: Jingchun Gao <gaojingchun1@huawei.com>
Signed-off-by: zhenwenqi2024 <zhenwenqi_2022@qq.com>
Co-authored-by: FENP <yuanyongjie.yyj@antgroup.com>
Co-authored-by: LookAround <lixushi@huawei.com>
Co-authored-by: Jingchun Gao <gaojingchun1@huawei.com>
Co-authored-by: zhenwenqi2024 <zhenwenqi_2022@qq.com>
Co-authored-by: Jingchun Gao <63247409+gjc0824@users.noreply.github.com>

2fd893b4

[Doc]: fix typos in various files (#28945) · 7ed27f3c
Didier Durand authored Nov 19, 2025
```
Signed-off-by: Didier Durand <durand.didier@gmail.com>
```
7ed27f3c

18 Nov, 2025 1 commit
- [Bugfix] Fix wrong CLI defaults for dynamic `SchedulerConfig` fields (#28872) · bf9e1e87
  Cyrus Leung authored Nov 18, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  bf9e1e87
16 Nov, 2025 1 commit

[V1] Support MP Executor for multi node distributed inference (#23691) · b316ac65

Lucia Fang authored Nov 16, 2025


Signed-off-by: Lu Fang <fanglu@fb.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Lucia Fang <fanglu@fb.com>
Signed-off-by: Lucia Fang <116399278+luccafong@users.noreply.github.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>

b316ac65

15 Nov, 2025 1 commit
- [RL] [V1] Remove unused device argument from reset_kv_cache (#28766) · dd6ac1c2
  Zhuohan Li authored Nov 14, 2025
```
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
```
  dd6ac1c2
14 Nov, 2025 2 commits

[DisaggEverything] Tokens in<>out `/generate` endpoint (#24261) · 6f1e7f72

Nicolò Lucchesi authored Nov 14, 2025

Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

6f1e7f72

[Config] Clean up SchedulerConfig initialization (#28665) · 511a6b61
Cyrus Leung authored Nov 14, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
511a6b61

13 Nov, 2025 2 commits
- [Perf] Support stream interval for reducing host overhead (#27869) · 5d6ce2b9
  elvischenv authored Nov 14, 2025
```
Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
```
  5d6ce2b9
- [BugFix] Ensure `EngineArgs.create_engine_config` is idempotent (#28515) · 327c0a9a
  Nick Hill authored Nov 13, 2025
```
Signed-off-by: Nick Hill <nhill@redhat.com>
```
  327c0a9a
12 Nov, 2025 2 commits

[Core] Encoder separation for Encode-Prefill-Decode Disaggregation (#25233) · 4ccffe56

Chenguang Zheng authored Nov 12, 2025


Signed-off-by: n00909098 <nguyen.kha.long@huawei.com>
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Signed-off-by: herotai214 <herotai214@gmail.com>
Signed-off-by: Khuong Le <khuong.le.manh@huawei.com>
Signed-off-by: Khuong Le <lemanhkhuong2611@gmail.com>
Co-authored-by: n00909098 <nguyen.kha.long@huawei.com>
Co-authored-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Co-authored-by: herotai214 <herotai214@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Khuong Le <khuong.le.manh@huawei.com>
Co-authored-by: Khuong Le <lemanhkhuong2611@gmail.com>

4ccffe56

[CPU] Refactor CPU attention backend (#27954) · 7f829be7
Li, Jiang authored Nov 12, 2025
```
Signed-off-by: jiang1.li <jiang1.li@intel.com>
```
7f829be7

11 Nov, 2025 1 commit

[Attention] Refactor CUDA attention backend selection logic (#24794) · b30dfa03

Matthew Bonanni authored Nov 11, 2025


Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni001@gmail.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

b30dfa03