Commits · 3de6e6a30e33406c2b3bb81fb5a82a2966cebd87 · OpenDAS / vllm_cscc

03 Jul, 2024 11 commits
- [core][distributed] support n layers % pp size != 0 (#6115) · 3de6e6a3
  youkaichao authored Jul 03, 2024
  
  3de6e6a3
- [ Misc ] Clean Up `CompressedTensorsW8A8` (#6113) · 62963d12
  Robert Shaw authored Jul 03, 2024
  
  62963d12
- [vlm] Remove vision language config. (#6089) · d9e98f42
  xwjiang2010 authored Jul 03, 2024
```
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
```
  d9e98f42
- [core][distributed] custom allreduce when pp size > 1 (#6117) · 3c6325f0
  youkaichao authored Jul 03, 2024
  
  3c6325f0
- [Kernel] Expand FP8 support to Ampere GPUs using FP8 Marlin (#5975) · 47f0954a
  Michael Goin authored Jul 03, 2024
  
  47f0954a
- [Bugfix] Fix `compute_logits` in Jamba (#6093) · 7cd2ebb0
  Roger Wang authored Jul 03, 2024
  
  7cd2ebb0
- [VLM][Frontend] Proper Image Prompt Formatting from OpenAI API (#6091) · 3a86b54f
  Roger Wang authored Jul 02, 2024
```
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
```
  3a86b54f
- [misc][distributed] error on invalid state (#6092) · f6662071
  youkaichao authored Jul 02, 2024
  
  f6662071
- [BugFix] Avoid unnecessary Ray import warnings (#6079) · d830656a
  Nick Hill authored Jul 02, 2024
  
  d830656a
- [Core] Dynamic image size support for VLMs (#5276) · 9831aec4
  Cyrus Leung authored Jul 03, 2024
```
Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>
Co-authored-by: Xiaowei Jiang <xwjiang2010@gmail.com>
Co-authored-by: ywang96 <ywang@roblox.com>
Co-authored-by: xwjiang2010 <87673679+xwjiang2010@users.noreply.github.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
```
  9831aec4
- [hardware][misc] introduce platform abstraction (#6080) · 482045ee
  youkaichao authored Jul 02, 2024
  
  482045ee
02 Jul, 2024 8 commits

[Model] Jamba support (#4115) · 9d6a8daa

Mor Zusman authored Jul 03, 2024


Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
Co-authored-by: Erez Schwartz <erezs@ai21.com>
Co-authored-by: Mor Zusman <morz@ai21.com>
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com>
Co-authored-by: Tomer Asida <tomera@ai21.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>

9d6a8daa

[CORE] Quantized lm-head Framework (#4442) · ee93f4f9
Qubitium-ModelCloud authored Jul 03, 2024
```
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
Co-authored-by: ZX <zx@lbx.dev>
```
ee93f4f9

[ Misc ] Refactor MoE to isolate Fp8 From Mixtral (#5970) · 7c008c51

Robert Shaw authored Jul 02, 2024


Co-authored-by: Robert Shaw <rshaw@neuralmagic>
Co-authored-by: Michael Goin <michael@neuralmagic.com>

7c008c51

[Core] Pipeline Parallel Support (#4412) · c5832d2a
Murali Andoorveedu authored Jul 02, 2024
```
Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
```
c5832d2a
[Speculative Decoding] MLPSpeculator Tensor Parallel support (1/2) (#6050) · 15aba081
Sirej Dua authored Jul 02, 2024
```
Co-authored-by: Sirej Dua <sirej.dua@databricks.com>
Co-authored-by: Sirej Dua <Sirej Dua>
```
15aba081

[VLM] Remove `image_input_type` from VLM config (#5852) · 98d6682c

xwjiang2010 authored Jul 02, 2024


Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>

98d6682c

[Frontend] Add template related params to request (#5709) · 2c37540a
danieljannai21 authored Jul 02, 2024

2c37540a
[Core] Optimize block_manager_v2 vs block_manager_v1 (to make V2 default) (#5602) · 3476ed08
Alexander Matveev authored Jul 01, 2024

3476ed08

01 Jul, 2024 7 commits
- [Model] Changes to MLPSpeculator to support tie_weights and input_scale (#5965) · 54600709
  Thomas Parnell authored Jul 02, 2024
```
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Co-authored-by: Joshua Rosenkranz <jmrosenk@us.ibm.com>
```
  54600709
- [BugFix] Ensure worker model loop is always stopped at the right time (#5987) · c87ebc3e
  Nick Hill authored Jul 01, 2024
  
  c87ebc3e
- [Bugfix] Add explicit `end_forward` calls to flashinfer (#6044) · c4059ea5
  Antoni Baum authored Jul 01, 2024
  
  c4059ea5
- [Bugfix] adding chunking mechanism to fused_moe to handle large inputs (#6029) · 12a59959
  Avshalom Manevich authored Jul 02, 2024
  
  12a59959
- [Bugfix] Use RayActorError for older versions of Ray in RayTokenizerGroupPool (#6039) · dec6fc6f
  Antoni Baum authored Jul 01, 2024
  
  dec6fc6f
- [Speculative Decoding 2/2 ] Integrate typical acceptance sampler into Spec Decode Worker (#5348) · 80ca1e6a
  sroy745 authored Jul 01, 2024
  
  80ca1e6a
- [misc][cuda] use nvml to avoid accidentally cuda initialization (#6007) · 614aa512
  youkaichao authored Jun 30, 2024
  
  614aa512
30 Jun, 2024 4 commits
- [ Misc ] Refactor w8a8 to use `process_weights_after_load` (Simplify Weight Loading) (#5940) · af9ad46f
  Robert Shaw authored Jun 30, 2024
```
Co-authored-by: Robert Shaw <rshaw@neuralmagic>
```
  af9ad46f
- [Misc] Fix `get_min_capability` (#5971) · 7836fdcc
  Dipika Sikka authored Jun 30, 2024
  
  7836fdcc
- [Lora] Use safetensor keys instead of adapter_config.json to find unexpected modules. (#5909) · f5e73c9f
  SangBin Cho authored Jul 01, 2024
```
Co-authored-by: sang <sangcho@anyscale.com>
```
  f5e73c9f
- [Frontend]: Support base64 embedding (#5935) · c6c240aa
  llmpros authored Jun 30, 2024
```
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
```
  c6c240aa
29 Jun, 2024 9 commits
- [CI/Build] Add TP test for vision models (#5892) · 99397da5
  Cyrus Leung authored Jun 29, 2024
  
  99397da5
- [ CI/Build ] Added E2E Test For Compressed Tensors (#5839) · 8dbfcd35
  Robert Shaw authored Jun 29, 2024
```
Co-authored-by: Michael Goin <michael@neuralmagic.com>
Co-authored-by: Robert Shaw <rshaw@neuralmagic>
```
  8dbfcd35
- [Kernel] Raise an exception in MoE kernel if the batch size is larger then 65k (#5939) · f7dac83d
  Cody Yu authored Jun 29, 2024
  
  f7dac83d
- [Core] Optimize `SequenceStatus.is_finished` by switching to IntEnum (#5974) · 7c01f706
  Antoni Baum authored Jun 29, 2024
  
  7c01f706
- [Bugfix] Support `eos_token_id` from `config.json` (#5954) · 51e971d3
  Cyrus Leung authored Jun 29, 2024
  
  51e971d3
- [Bugfix] Fix precisions in Gemma 1 (#5913) · 580353da
  Woosuk Kwon authored Jun 28, 2024
  
  580353da
- [Misc] Extend vLLM Metrics logging API (#5925) · 906a19cd
  William Lin authored Jun 28, 2024
```
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
```
  906a19cd
- [Bugfix][TPU] Fix pad slot id (#5977) · 7f83f40d
  Woosuk Kwon authored Jun 28, 2024
  
  7f83f40d
- [Bugfix][TPU] Fix TPU sampler output (#5978) · 54814fd8
  Woosuk Kwon authored Jun 28, 2024
  
  54814fd8
28 Jun, 2024 1 commit
- [Kernel] Flashinfer for prefill & decode, with Cudagraph support for decode (#4628) · 7041de43
  Lily Liu authored Jun 28, 2024
```
Co-authored-by: LiuXiaoxuanPKU &lt;llilyliupku@gmail.com&gt;, bong-furiosa <bongwon.jang@furiosa.ai>
```
  7041de43