Commits · c0724fc9150329d42abaf2f0f77dc8ca91d48acb · OpenDAS / vllm_cscc

13 May, 2024 1 commit
- [Scheduler] Warning upon preemption and Swapping (#4647) · e7c46b95
  SangBin Cho authored May 13, 2024
```
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
```
  e7c46b95
11 May, 2024 1 commit
- [Model][Misc] Add e5-mistral-7b-instruct and Embedding API (#3734) · e254497b
  Chang Su authored May 11, 2024
  
  e254497b
08 May, 2024 1 commit
- [Core][Optimization] change python dict to pytorch tensor for blocks to swap (#4659) · 20cfcdec
  youkaichao authored May 08, 2024
  
  20cfcdec
07 May, 2024 2 commits
- [Core][Optimization] change copy-on-write from dict[int, list] to list (#4648) · 469f85c7
  youkaichao authored May 07, 2024
  
  469f85c7
- [Core][Optimization] change python dict to pytorch tensor (#4607) · 63575bc2
  youkaichao authored May 06, 2024
  
  63575bc2
04 May, 2024 1 commit
- [Misc][Refactor] Introduce ExecuteModelData (#4540) · bc8ad684
  Cody Yu authored May 03, 2024
  
  bc8ad684
02 May, 2024 2 commits
- [Core] Ignore infeasible swap requests. (#4557) · 0f8a9140
  SangBin Cho authored May 03, 2024
  
  0f8a9140
- [Bug fix][Core] assert num_new_tokens == 1 fails when SamplingParams.n is not... · 0d62fe58
  SangBin Cho authored May 02, 2024
```
[Bug fix][Core] assert num_new_tokens == 1 fails when SamplingParams.n is not 1 and max_tokens is large & Add tests for preemption (#4451)
```
  0d62fe58
28 Apr, 2024 1 commit

Add more Prometheus metrics (#2764) · bf480c53

Ronen Schaffer authored Apr 29, 2024


Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>

bf480c53

26 Apr, 2024 2 commits
- [Core] Refactoring sampler and support prompt logprob for chunked prefill (#4309) · 603ad848
  SangBin Cho authored Apr 26, 2024
  
  603ad848
- [CI] Disable non-lazy string operation on logging (#4326) · a88081bf
  SangBin Cho authored Apr 26, 2024
```
Co-authored-by: Danny Guinther <dguinther@neuralmagic.com>
```
  a88081bf
23 Apr, 2024 1 commit
- [Core] Scheduling optimization 2 (#4280) · 050f285f
  SangBin Cho authored Apr 23, 2024
  
  050f285f
22 Apr, 2024 1 commit
- [Core] Scheduler perf fix (#4270) · ad8d696a
  SangBin Cho authored Apr 23, 2024
  
  ad8d696a
16 Apr, 2024 1 commit
- [Speculative decoding 6/9] Integrate speculative decoding with LLMEngine (#3894) · e95cd879
  Cade Daniel authored Apr 16, 2024
  
  e95cd879
12 Apr, 2024 2 commits
- [mypy] Add mypy type annotation part 1 (#4006) · 09473ee4
  SangBin Cho authored Apr 13, 2024
  
  09473ee4
- [Misc] Fix typo in scheduler.py (#4022) · d4ec9ffb
  Zhuohan Li authored Apr 12, 2024
  
  d4ec9ffb
11 Apr, 2024 1 commit
- [Core][5/N] Fully working chunked prefill e2e (#3884) · 67b4221a
  SangBin Cho authored Apr 11, 2024
  
  67b4221a
05 Apr, 2024 1 commit
- [Chunked Prefill][4/n] Chunked prefill scheduler. (#3853) · 18de8834
  SangBin Cho authored Apr 06, 2024
  
  18de8834
03 Apr, 2024 1 commit
- [3/N] Refactor scheduler for chunked prefill scheduling (#3550) · 3dcb3e8b
  SangBin Cho authored Apr 04, 2024
  
  3dcb3e8b
01 Apr, 2024 1 commit
- [Speculative decoding 4/9] Lookahead scheduling for speculative decoding (#3250) · 93deb0b3
  Cade Daniel authored Apr 01, 2024
  
  93deb0b3
28 Mar, 2024 2 commits
- [2/N] Chunked prefill data update (#3538) · b51c1cc9
  SangBin Cho authored Mar 29, 2024
  
  b51c1cc9
- [Core][Bugfix]Refactor block manager for better testability (#3492) · 14ccd94c
  Cade Daniel authored Mar 27, 2024
  
  14ccd94c
25 Mar, 2024 2 commits
- [Feature] Add vision language model support. (#3042) · 64172a97
  xwjiang2010 authored Mar 25, 2024
  
  64172a97
- [CI] Try introducing isort. (#3495) · 01bfb22b
  SangBin Cho authored Mar 25, 2024
  
  01bfb22b
22 Mar, 2024 1 commit
- Dynamic scheduler delay to improve ITL performance (#3279) · cf2f084d
  Thomas Parnell authored Mar 22, 2024
```
Co-authored-by: Jan van Lunteren <jvl@zurich.ibm.com>
```
  cf2f084d
20 Mar, 2024 1 commit
- [1/n][Chunked Prefill] Refactor input query shapes (#3236) · 6e435de7
  SangBin Cho authored Mar 21, 2024
  
  6e435de7
15 Mar, 2024 1 commit
- Fixes the misuse/mixuse of time.time()/time.monotonic() (#3220) · 14b8ae02
  Tao He authored Mar 16, 2024
```
Signed-off-by: Tao He <sighingnow@gmail.com>
Co-authored-by: simon-mo <simon.mo@hey.com>
```
  14b8ae02
11 Mar, 2024 1 commit
- Re-enable the 80 char line width limit (#3305) · 2f8844ba
  Zhuohan Li authored Mar 10, 2024
  
  2f8844ba
05 Mar, 2024 1 commit
- Store `eos_token_id` in `Sequence` for easy access (#3166) · 8999ec3c
  Nick Hill authored Mar 05, 2024
  
  8999ec3c
02 Mar, 2024 1 commit

Add Automatic Prefix Caching (#2762) · ce4f5a29

Sage Moore authored Mar 02, 2024


Co-authored-by: ElizaWszola <eliza@neuralmagic.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>

ce4f5a29

22 Feb, 2024 1 commit
- chore(vllm): codespell for spell checking (#2820) · 93dc5a28
  Massimiliano Pronesti authored Feb 22, 2024
  
  93dc5a28
21 Feb, 2024 2 commits
- Support per-request seed (#2514) · 7d2dcce1
  Nick Hill authored Feb 21, 2024
  
  7d2dcce1
- Add metrics to RequestOutput (#2876) · 017d9f15
  Antoni Baum authored Feb 20, 2024
  
  017d9f15
23 Jan, 2024 1 commit

[Experimental] Add multi-LoRA support (#1804) · 9b945daa

Antoni Baum authored Jan 24, 2024


Co-authored-by: Chen Shen <scv119@gmail.com>
Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com>
Co-authored-by: Avnish Narayan <avnish@anyscale.com>

9b945daa

21 Jan, 2024 1 commit
- [Fix] Keep `scheduler.running` as deque (#2523) · d75c4073
  Nick Hill authored Jan 20, 2024
  
  d75c4073
18 Jan, 2024 2 commits
- [BugFix] Fix abort_seq_group (#2463) · d2a68364
  ljss authored Jan 19, 2024
  
  d2a68364
- [Experimental] Prefix Caching Support (#1669) · d10f8e1d
  shiyi.c_98 authored Jan 17, 2024
```
Co-authored-by: DouHappy <2278958187@qq.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
```
  d10f8e1d
12 Jan, 2024 2 commits
- fix: deque mutated during iteration in abort_seq_group (#2371) · 48cf1e41
  陈序 authored Jan 13, 2024
  
  48cf1e41
- [DOC] Add additional comments for LLMEngine and AsyncLLMEngine (#1011) · 6549aef2
  Jiaxiang authored Jan 12, 2024
  
  6549aef2
07 Jan, 2024 1 commit
- Changed scheduler to use deques instead of lists (#2290) · 05921a9a
  Nadav Shmayovits authored Jan 07, 2024
```
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  05921a9a