Skip to content

GitLab

Switch branch/tag
  1. 09 Dec, 2025 1 commit
  3. 03 Dec, 2025 1 commit
  5. 01 Dec, 2025 1 commit
    • shivampr's avatar
      [Core][Observability] Add KV cache residency metrics (#27793) · cabc77cc
      shivampr authored 
      

Introduces three new Prometheus histograms for fine-grained observability of KV cache residency behavior:

vllm:kv_block_lifetime_seconds — total lifetime from allocation to free
vllm:kv_block_idle_before_evict_seconds — idle duration before eviction
vllm:kv_block_reuse_gap_seconds — time between consecutive reuses of the same block

These metrics help operators analyze KV cache efficiency, reuse patterns, and eviction timing beyond simple utilization rates.

Implementation uses monotonic timestamps for accuracy, 1% sampling for minimal overhead (~48 bytes/block), and is fully thread-safe with zero runtime cost when disabled.

Two new runtime flags are introduced:

--kv-cache-metrics – enable KV cache residency metrics
--kv-cache-metrics-sample – control sampling ratio (default: 0.01)
Signed-off-by: default avatarShivam <shivamprasad91@gmail.com>
      cabc77cc
  7. 10 Nov, 2025 1 commit
  9. 05 Nov, 2025 1 commit
  11. 23 Oct, 2025 1 commit
  13. 12 Oct, 2025 1 commit
  15. 10 Oct, 2025 2 commits
  17. 05 Oct, 2025 2 commits
  19. 27 Sep, 2025 1 commit
  21. 24 Sep, 2025 1 commit
  23. 19 Sep, 2025 1 commit
  25. 12 Sep, 2025 1 commit
  27. 02 Sep, 2025 2 commits
  29. 02 Aug, 2025 1 commit
  31. 20 Jun, 2025 1 commit
  33. 19 Jun, 2025 1 commit
  35. 14 Jun, 2025 1 commit
  37. 03 Jun, 2025 1 commit
  39. 12 May, 2025 1 commit
  41. 01 Apr, 2025 1 commit
  43. 24 Mar, 2025 1 commit
  45. 19 Mar, 2025 1 commit
  47. 07 Mar, 2025 1 commit
  49. 03 Mar, 2025 3 commits
  51. 27 Feb, 2025 1 commit
  53. 25 Feb, 2025 1 commit
  55. 11 Feb, 2025 2 commits
  57. 07 Feb, 2025 1 commit
    • afeldman-nm's avatar
      [V1] Logprobs and prompt logprobs support (#9880) · 0630d453
      afeldman-nm authored 
      

This PR is adding support for sample logprobs & prompt logprobs to vLLM v1.

New behavior:

- During model execution, model runner computes sample logprobs (if user-provided logprobs setting is not None) and prompt logprobs (if user-provided prompt_logprobs setting is not None). For both sample and prompt logprobs, the engine core returns 3 vectors: token ids, token logprob values, token ranks. Ranks reflect tokens' 1-indexed positions in the vocabulary vector after sorting the vocabulary by log probability in descending order.
- In scheduler.update_from_output(), sample and prompt logprobs are incorporated into the EngineCoreOutput data structure which is transferred to the engine client. If multiprocessing is enabled, then sample and prompt logprobs will be (de)serialized when the EngineCoreOutput data structure is (de)serialized.
- During output processing, the LogprobsProcessor transforms the triplet of token ids, token logprobs values, and token ranks into the OpenAI-compatible List[Dict[token id,Logprob]] format (for sample and prompt logprobs respectively.)
- Each Logprob instance (whether sample- or prompt-) consists of a token's log-probability, rank, and detokenized string representation. Note that logprob detokenization is handled by the LogprobsProcessor not the detokenizer.
Signed-off-by: default avatarAndrew Feldman <afeldman@neuralmagic.com>
Signed-off-by: default avatarNick Hill <nhill@redhat.com>
Signed-off-by: default avatarrshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: default avatarrshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: default avatarNick Hill <nhill@redhat.com>
      0630d453
  59. 05 Feb, 2025 2 commits
  61. 02 Feb, 2025 1 commit
    • Russell Bryant's avatar
      [Misc] Add SPDX-License-Identifier headers to python source files (#12628) · e489ad7a
      Russell Bryant authored 
      - **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/

Signed-off-by: default avatarRussell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
Signed-off-by: default avatarRussell Bryant <rbryant@redhat.com>

---------
Signed-off-by: default avatarRussell Bryant <rbryant@redhat.com>
      e489ad7a
  63. 30 Jan, 2025 1 commit
  65. 29 Jan, 2025 1 commit