Skip to content

GitLab

Switch branch/tag
  1. 11 Feb, 2025 3 commits
  3. 10 Feb, 2025 1 commit
  5. 08 Feb, 2025 7 commits
  7. 07 Feb, 2025 1 commit
    • afeldman-nm's avatar
      [V1] Logprobs and prompt logprobs support (#9880) · 0630d453
      afeldman-nm authored 
      

This PR is adding support for sample logprobs & prompt logprobs to vLLM v1.

New behavior:

- During model execution, model runner computes sample logprobs (if user-provided logprobs setting is not None) and prompt logprobs (if user-provided prompt_logprobs setting is not None). For both sample and prompt logprobs, the engine core returns 3 vectors: token ids, token logprob values, token ranks. Ranks reflect tokens' 1-indexed positions in the vocabulary vector after sorting the vocabulary by log probability in descending order.
- In scheduler.update_from_output(), sample and prompt logprobs are incorporated into the EngineCoreOutput data structure which is transferred to the engine client. If multiprocessing is enabled, then sample and prompt logprobs will be (de)serialized when the EngineCoreOutput data structure is (de)serialized.
- During output processing, the LogprobsProcessor transforms the triplet of token ids, token logprobs values, and token ranks into the OpenAI-compatible List[Dict[token id,Logprob]] format (for sample and prompt logprobs respectively.)
- Each Logprob instance (whether sample- or prompt-) consists of a token's log-probability, rank, and detokenized string representation. Note that logprob detokenization is handled by the LogprobsProcessor not the detokenizer.
Signed-off-by: default avatarAndrew Feldman <afeldman@neuralmagic.com>
Signed-off-by: default avatarNick Hill <nhill@redhat.com>
Signed-off-by: default avatarrshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: default avatarrshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: default avatarNick Hill <nhill@redhat.com>
      0630d453
  9. 06 Feb, 2025 2 commits
  11. 05 Feb, 2025 3 commits
  13. 04 Feb, 2025 2 commits
  15. 03 Feb, 2025 1 commit
  17. 02 Feb, 2025 3 commits
  19. 01 Feb, 2025 1 commit
  21. 31 Jan, 2025 1 commit
    • Chen Zhang's avatar
      [v1][Bugfix] Add extra_keys to block_hash for prefix caching (#12603) · 89003c40
      Chen Zhang authored 
      

This pr adds extra key to block hash, to generate different hash value
for two blocks with the same token string but different extra_keys in
their parent blocks. For example, it can generate different hash value
for the second block of the following two requests:
```python
request1 = make_request(
        request_id=0,
        prompt_token_ids=[_ for _ in range(6)],
        mm_positions=[{
            "offset": 0,
            "length": 3
        }, {
            "offset": 3,
            "length": 3
        }],
        mm_hashes=["hash1", "hash2"],
    )
    request2 = make_request(
        request_id=1,
        prompt_token_ids=[_ for _ in range(6)],
        mm_positions=[{
            "offset": 0,
            "length": 3
        }, {
            "offset": 3,
            "length": 3
        }],
        mm_hashes=["hash3", "hash2"],
    )
```

---------
Signed-off-by: default avatarChen Zhang <zhangch99@outlook.com>
      89003c40
  23. 30 Jan, 2025 2 commits
  25. 29 Jan, 2025 2 commits
  27. 28 Jan, 2025 4 commits
  29. 27 Jan, 2025 3 commits
  31. 26 Jan, 2025 2 commits
  33. 24 Jan, 2025 2 commits