Commits · eb6d3c264d0cd8e44dec16bca7947fbe96415ce9 · OpenDAS / vllm_cscc

22 May, 2024 1 commit
- [Core] Eliminate parallel worker per-step task scheduling overhead (#4894) · eb6d3c26
  Nick Hill authored May 22, 2024
  
  eb6d3c26
16 May, 2024 2 commits
- [Speculative decoding][Re-take] Enable TP>1 speculative decoding (#4840) · 973617ae
  Cody Yu authored May 16, 2024
```
Co-authored-by: Cade Daniel <edacih@gmail.com>
Co-authored-by: Cade Daniel <cade@anyscale.com>
```
  973617ae
- [Core] Implement sharded state loader (#4690) · 30e75439
  Aurick Qiao authored May 16, 2024
```
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  30e75439
14 May, 2024 1 commit
- [Core] Add MultiprocessingGPUExecutor (#4539) · 676a9998
  Nick Hill authored May 14, 2024
```
Co-authored-by: SAHIL SUNEJA <suneja@us.ibm.com>
```
  676a9998
11 May, 2024 1 commit
- [Model][Misc] Add e5-mistral-7b-instruct and Embedding API (#3734) · e254497b
  Chang Su authored May 11, 2024
  
  e254497b
08 May, 2024 1 commit
- [Dynamic Spec Decoding] Auto-disable by the running queue size (#4592) · f942efb5
  Cody Yu authored May 08, 2024
```
Co-authored-by: Cade Daniel <edacih@gmail.com>
```
  f942efb5
07 May, 2024 1 commit

[Bug fix][Core] fixup ngram not setup correctly (#4551) · 8344f774

leiwen83 authored May 08, 2024


Co-authored-by: Lei Wen <wenlei03@qiyi.com>
Co-authored-by: Cade Daniel <edacih@gmail.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>

8344f774

04 May, 2024 1 commit
- [Misc][Refactor] Introduce ExecuteModelData (#4540) · bc8ad684
  Cody Yu authored May 03, 2024
  
  bc8ad684
02 May, 2024 1 commit
- [Misc] centralize all usage of environment variables (#4548) · 5b8a7c1c
  youkaichao authored May 02, 2024
  
  5b8a7c1c
01 May, 2024 3 commits
- [Core] Add `multiproc_worker_utils` for multiprocessing-based workers (#4357) · a657bfc4
  Nick Hill authored May 01, 2024
  
  a657bfc4
- [Speculative decoding] Add ngram prompt lookup decoding (#4237) · b38e42fb
  leiwen83 authored May 02, 2024
```
Co-authored-by: Lei Wen <wenlei03@qiyi.com>
```
  b38e42fb
- [Core] Centralize GPU Worker construction (#4419) · 2e240c69
  Nick Hill authored Apr 30, 2024
  
  2e240c69
30 Apr, 2024 1 commit
- [BugFix] fix num_lookahead_slots missing in async executor (#4165) · 4bb53e2d
  leiwen83 authored May 01, 2024
```
Co-authored-by: Lei Wen <wenlei03@qiyi.com>
```
  4bb53e2d
27 Apr, 2024 2 commits
- [BugFix] Fix return type of executor execute_model methods (#4402) · ba4be44c
  Nick Hill authored Apr 27, 2024
  
  ba4be44c
- [Core] Introduce `DistributedGPUExecutor` abstract class (#4348) · 258a2c58
  Nick Hill authored Apr 26, 2024
  
  258a2c58
26 Apr, 2024 1 commit
- [CI] Disable non-lazy string operation on logging (#4326) · a88081bf
  SangBin Cho authored Apr 26, 2024
```
Co-authored-by: Danny Guinther <dguinther@neuralmagic.com>
```
  a88081bf
25 Apr, 2024 2 commits
- [Core] Add `shutdown()` method to `ExecutorBase` (#4349) · 15e7c675
  Nick Hill authored Apr 25, 2024
  
  15e7c675
- [Core] Move ray_utils.py from `engine` to `executor` package (#4347) · 479d69fa
  Nick Hill authored Apr 24, 2024
  
  479d69fa
23 Apr, 2024 3 commits
- [Bugfix] Add init_cached_hf_modules to RayWorkerWrapper (#4286) · d87f39e9
  DefTruth authored Apr 24, 2024
  
  d87f39e9
- [Speculative decoding 7/9] Speculative decoding end-to-end correctness tests. (#3951) · 62b8aebc
  Cade Daniel authored Apr 23, 2024
  
  62b8aebc
- [Core] Some simplification of WorkerWrapper changes (#4183) · 8f2ea22b
  Nick Hill authored Apr 23, 2024
  
  8f2ea22b
22 Apr, 2024 2 commits
- [Frontend] Enable support for CPU backend in AsyncLLMEngine. (#3993) · 077f0a2e
  Tao He authored Apr 22, 2024
```
Signed-off-by: Tao He <sighingnow@gmail.com>
```
  077f0a2e
- [Misc] Add vision language model support to CPU backend (#3968) · 296cdf8a
  Isotr0py authored Apr 22, 2024
  
  296cdf8a
18 Apr, 2024 2 commits
- [Core] add an option to log every function call to for debugging hang/crash in... · 8a7a3e44
  youkaichao authored Apr 18, 2024
```
[Core] add an option to log every function call to for debugging hang/crash in distributed inference (#4079)
Co-authored-by: Simon Mo <simon.mo@hey.com>
```
  8a7a3e44
- [CI/CD] add neuron docker and ci test scripts (#3571) · cd2f63fb
  Liangfu Chen authored Apr 18, 2024
  
  cd2f63fb
17 Apr, 2024 2 commits
- [Core] RayWorkerVllm --> WorkerWrapper to reduce duplication (#4024) · 8438e056
  youkaichao authored Apr 17, 2024
```
[Core] replace narrow-usage RayWorkerVllm to general WorkerWrapper to reduce code duplication (#4024)
```
  8438e056
- [Misc] [CI] Fix CI failure caught after merge (#4126) · d150e4f8
  Cade Daniel authored Apr 16, 2024
  
  d150e4f8
16 Apr, 2024 2 commits
- [Speculative decoding 6/9] Integrate speculative decoding with LLMEngine (#3894) · e95cd879
  Cade Daniel authored Apr 16, 2024
  
  e95cd879
- [Core] Refactor model loading code (#4097) · 69e1d2fb
  Antoni Baum authored Apr 16, 2024
  
  69e1d2fb
15 Apr, 2024 3 commits
- [Bugfix] Fix ray workers profiling with nsight (#4095) · 4695397d
  Ricky Xu authored Apr 15, 2024
  
  4695397d
- [Core] Simplifications to executor classes (#4071) · eb46fbfd
  Nick Hill authored Apr 15, 2024
  
  eb46fbfd
- [Misc][Minor] Fix CPU block num log in CPUExecutor. (#4088) · 0003e915
  Li, Jiang authored Apr 15, 2024
  
  0003e915
14 Apr, 2024 1 commit
- [Frontend] [Core] feat: Add model loading using `tensorizer` (#3476) · 711a0002
  Sanger Steel authored Apr 13, 2024
  
  711a0002
13 Apr, 2024 2 commits
- [Bugfix] fix_small_bug_in_neuron_executor (#4051) · 0a430b4a
  zspo authored Apr 13, 2024
  
  0a430b4a
- [Bugfix] More type hint fixes for py 3.8 (#4039) · 5c2e66e4
  Dylan Hawk authored Apr 12, 2024
  
  5c2e66e4
12 Apr, 2024 2 commits
- [mypy] Add mypy type annotation part 1 (#4006) · 09473ee4
  SangBin Cho authored Apr 13, 2024
  
  09473ee4
- [Bugfix] fix type hint for py 3.8 (#4036) · 96b6a6d7
  youkaichao authored Apr 12, 2024
  
  96b6a6d7
11 Apr, 2024 1 commit
- [Hardware][Intel] Isolate CPUModelRunner and ModelRunner for better maintenance (#3824) · 8afca508
  bigPYJ1151 authored Apr 12, 2024
  
  8afca508
09 Apr, 2024 1 commit
- [Misc] [Core] Implement RFC "Augment BaseExecutor interfaces to enable... · e7c7067b
  Cade Daniel authored Apr 09, 2024
```
[Misc] [Core] Implement RFC "Augment BaseExecutor interfaces to enable hardware-agnostic speculative decoding" (#3837)
```
  e7c7067b
07 Apr, 2024 1 commit
- [Bugfix] Fix Llava inference with Tensor Parallelism. (#3883) · 0ce0539d
  Isotr0py authored Apr 07, 2024
  
  0ce0539d