Commits · 6a62cb82ccace1209f7b8bbec95025e047a95ded · OpenDAS / vllm_cscc · GitLab

28 Jun, 2024 15 commits
- [Bugfix] Fix Engine Failing After Invalid Request - AsyncEngineDeadError (#5963) · 6a62cb82
  Robert Shaw authored Jun 28, 2024
```
Co-authored-by: Robert Shaw <rshaw@neuralmagic>
```
  6a62cb82
- Unmark more files as executable (#5962) · 5d2a1a9c
  Tyler Michael Smith authored Jun 28, 2024
  
  5d2a1a9c
- [Bugfix] Only add `Attention.kv_scale` if kv cache quantization is enabled (#5936) · 4bf35ed9
  Michael Goin authored Jun 28, 2024
  
  4bf35ed9
- Support Deepseek-V2 (#4650) · be0b3af9
  wangding zeng authored Jun 29, 2024
```
Co-authored-by: Philipp Moritz <pcmoritz@gmail.com>
```
  be0b3af9
- [ Bugfix ] Enabling Loading Models With Fused QKV/MLP on Disk with FP8 (#5921) · 2cd402e1
  Robert Shaw authored Jun 28, 2024
```
Co-authored-by: Robert Shaw <rshaw@neuralmagic>
```
  2cd402e1
- [ Misc ] Remove `fp8_shard_indexer` from Col/Row Parallel Linear (Simplify Weight Loading) (#5928) · b1852307
  Robert Shaw authored Jun 28, 2024
```
Co-authored-by: Robert Shaw <rshaw@neuralmagic>
```
  b1852307
- [Spec Decode] Introduce DraftModelRunner (#5799) · b2c62023
  Cody Yu authored Jun 28, 2024
  
  b2c62023
- [Distributed] Make it clear that % should not be in tensor dict keys. (#5927) · b90d8cd8
  xwjiang2010 authored Jun 28, 2024
```
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>
```
  b90d8cd8
- [Bugfix] Better error message for MLPSpeculator when `num_speculative_tokens`... · ec1ad004
  Thomas Parnell authored Jun 28, 2024
```
[Bugfix] Better error message for MLPSpeculator when `num_speculative_tokens` is set too high (#5894)
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
```
  ec1ad004
- [Hardware][Intel] OpenVINO vLLM backend (#5379) · 57f09a41
  Ilya Lavrenov authored Jun 28, 2024
  
  57f09a41
- Unmark fused_moe config json file as executable (#5960) · 59326344
  Tyler Michael Smith authored Jun 28, 2024
  
  59326344
- [Core] Registry for processing model inputs (#5214) · 5cbe8d15
  Cyrus Leung authored Jun 28, 2024
```
Co-authored-by: ywang96 <ywang@roblox.com>
```
  5cbe8d15
- [Bugfix][Hardware][Intel CPU] Fix unpassed multi_modal_kwargs for CPU runner (#5956) · 0d0e3a42
  Isotr0py authored Jun 28, 2024
  
  0d0e3a42
- [VLM][BugFix] Make sure that `multi_modal_kwargs` can broadcast properly with ring buffer. (#5905) · 74d55c06
  xwjiang2010 authored Jun 28, 2024
```
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
```
  74d55c06
- [Hardware][TPU] Optimize KV cache swapping (#5878) · f136da15
  Woosuk Kwon authored Jun 27, 2024
  
  f136da15
27 Jun, 2024 10 commits
- [Kernel][ROCm][AMD] fused_moe Triton configs v2 for mi300X (#5932) · c3dde367
  Divakar Verma authored Jun 27, 2024
  
  c3dde367
- [core][misc] remove logical block (#5882) · 64e8d2a7
  youkaichao authored Jun 27, 2024
  
  64e8d2a7
- [Model] Add Gemma 2 (#5908) · 79c92c7c
  Woosuk Kwon authored Jun 27, 2024
  
  79c92c7c
- [BugFix] Fix `min_tokens` behaviour for multiple eos tokens (#5849) · 365791ff
  Nick Hill authored Jun 27, 2024
  
  365791ff
- [BugFix] Fix `MLPSpeculator` handling of `num_speculative_tokens` (#5876) · 691e29ec
  Nick Hill authored Jun 27, 2024
  
  691e29ec
- [Model][Bugfix] Implicit model flags and reenable Phi-3-Vision (#5896) · 98cf2ed6
  Cyrus Leung authored Jun 28, 2024
  
  98cf2ed6
- [Bugfix] Fix img_sizes Parsing in Phi3-Vision (#5888) · 2061f0b8
  Roger Wang authored Jun 27, 2024
  
  2061f0b8
- [Model] Add base class for LoRA-supported models (#5018) · 96354d6a
  Cyrus Leung authored Jun 27, 2024
  
  96354d6a
- [VLM][Bugfix] Make sure that `multi_modal_kwargs` is broadcasted properly (#5880) · d12af207
  xwjiang2010 authored Jun 27, 2024
```
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>
```
  d12af207
- [BugFix] Fix cuda graph for MLPSpeculator (#5875) · 2110557d
  Nick Hill authored Jun 26, 2024
```
Co-authored-by: Abhinav Goyal <abhinav.goyal@flipkart.com>
```
  2110557d
26 Jun, 2024 10 commits
- [Bugfix][TPU] Fix CPU cache allocation (#5869) · f5c8628f
  Woosuk Kwon authored Jun 26, 2024
  
  f5c8628f
- [Hardware][TPU] Support parallel sampling & Swapping (#5855) · cbc53b6b
  Woosuk Kwon authored Jun 26, 2024
  
  cbc53b6b
- [Frontend] Add tokenize/detokenize endpoints (#5054) · c54269d9
  sasha0552 authored Jun 26, 2024
  
  c54269d9
- [Kernel] Adding bias epilogue support for `cutlass_scaled_mm` (#5560) · 5bfd1bbc
  Luka Govedič authored Jun 26, 2024
```
Co-authored-by: Chih-Chieh-Yang <7364402+cyang49@users.noreply.github.com>
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
  5bfd1bbc
- [Bugfix][TPU] Fix KV cache size calculation (#5860) · 3439c5a8
  Woosuk Kwon authored Jun 26, 2024
  
  3439c5a8
- [Bugfix] Fix embedding to support 2D inputs (#5829) · 6806998b
  Woosuk Kwon authored Jun 26, 2024
  
  6806998b
- [bugfix][distributed] fix shm broadcast when the queue size is full (#5801) · 515080ad
  youkaichao authored Jun 25, 2024
  
  515080ad
- [Misc][Doc] Add Example of using OpenAI Server with VLM (#5832) · 3aa7b6cf
  Roger Wang authored Jun 25, 2024
  
  3aa7b6cf
- [Core] Refactor Worker and ModelRunner to consolidate control plane communication (#5408) · dda48115
  Stephanie Wang authored Jun 25, 2024
```
Signed-off-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Stephanie <swang@anyscale.com>
Co-authored-by: Stephanie <swang@anyscale.com>
```
  dda48115
- [Bugfix] Fix assertion in NeuronExecutor (#5841) · 82079729
  aws-patlange authored Jun 25, 2024
  
  82079729
25 Jun, 2024 5 commits
- [Hardware][TPU] Raise errors for unsupported sampling params (#5850) · f178e56c
  Woosuk Kwon authored Jun 25, 2024
  
  f178e56c
- [Hardware][AMD][CI/Build][Doc] Upgrade to ROCm 6.1, Dockerfile improvements, test fixes (#5422) · dd793d1d
  Matt Wong authored Jun 25, 2024
  
  dd793d1d
- [Hardware][TPU] Refactor TPU backend (#5831) · bc34937d
  Woosuk Kwon authored Jun 25, 2024
  
  bc34937d
- [Misc] Update `w4a16` `compressed-tensors` support to include `w8a16` (#5794) · dd248f76
  Dipika Sikka authored Jun 25, 2024
  
  dd248f76
- [Core] Add fault tolerance for `RayTokenizerGroupPool` (#5748) · 67882dbb
  Antoni Baum authored Jun 25, 2024
  
  67882dbb