Commits · d65049daabe9a80783b0547fd85dd39a18a905b3 · OpenDAS / vllm_cscc · GitLab

17 Oct, 2024 14 commits
- [Bugfix] Add random_seed to sample_hf_requests in benchmark_serving script (#9013) · d65049da
  Kai Wu authored Oct 17, 2024
```
Co-authored-by: Isotr0py <2037008807@qq.com>
```
  d65049da
- [Bugfix] Fix support for dimension like integers and ScalarType (#9299) · eca2c5f7
  bnellnm authored Oct 17, 2024
  
  eca2c5f7
- [torch.compile] Fine-grained CustomOp enabling mechanism (#9300) · 0f41fbe5
  Luka Govedič authored Oct 17, 2024
  
  0f41fbe5
- [Misc] Remove commit id file (#9470) · 7871659a
  Cyrus Leung authored Oct 18, 2024
  
  7871659a
- [CI/Build] remove .github from .dockerignore, add dirty repo check (#9375) · a2c71c54
  Daniele authored Oct 17, 2024
  
  a2c71c54
- [Core] Deprecating block manager v1 and make block manager v2 default (#8704) · 81ede99c
  Kuntai Du authored Oct 17, 2024
```
Removing the block manager v1. This is the initial piece of prefix-caching-centric design. In order to achieve prefix-caching-centric design, we need to simplify the code path so that we only use v2 block manager (which has much higher performance on prefix caching).
```
  81ede99c
- [Hardware][CPU] compressed-tensor INT8 W8A8 AZP support (#9344) · 5eda21e7
  Li, Jiang authored Oct 18, 2024
  
  5eda21e7
- [TPU] Call torch._sync(param) during weight loading (#9437) · 8e1cddcd
  Woosuk Kwon authored Oct 17, 2024
  
  8e1cddcd
- [Bugfix] Allow prefill of assistant response when using `mistral_common` (#9446) · 5e443b59
  sasha0552 authored Oct 17, 2024
  
  5e443b59
- [misc] CUDA Time Layerwise Profiler (#8337) · 9d30a056
  Lucas Wilkinson authored Oct 17, 2024
```
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
```
  9d30a056
- [Misc] Print stack trace using `logger.exception` (#9461) · 390be746
  Cyrus Leung authored Oct 17, 2024
  
  390be746
- [Kernel] Add Exllama as a backend for compressed-tensors (#9395) · e312e52b
  Lucas Wilkinson authored Oct 17, 2024
  
  e312e52b
- Add notes on the use of Slack (#9442) · dbfa8d31
  Yuan Tang authored Oct 17, 2024
  
  dbfa8d31
- [BugFix] [Kernel] Fix GPU SEGV occurring in int8 kernels (#9391) · 92d86da2
  rasmith authored Oct 16, 2024
  
  92d86da2
16 Oct, 2024 14 commits
- [Bugfix][Kernel] Prevent integer overflow in fp8 dynamic per-token quantize kernel (#9425) · c3fab5f7
  Tyler Michael Smith authored Oct 16, 2024
  
  c3fab5f7
- [CI/Build] mypy: Resolve some errors from checking vllm/engine (#9267) · 776dbd74
  Russell Bryant authored Oct 16, 2024
```
Signed-off-by: Russell Bryant <rbryant@redhat.com>
```
  776dbd74
- [Performance][Spec Decode] Optimize ngram lookup performance (#9333) · 83450458
  Lily Liu authored Oct 16, 2024
  
  83450458
- [Model][Bugfix] Add FATReLU activation and support for openbmb/MiniCPM-S-1B-sft (#9396) · 5b8a1fde
  Junhao Li authored Oct 16, 2024
  
  5b8a1fde
- [Kernel][Model] Improve continuous batching for Jamba and Mamba (#9189) · fb60ae9b
  Mor Zusman authored Oct 17, 2024
  
  fb60ae9b
- Support mistral interleaved attn (#9414) · 415f76a9
  Patrick von Platen authored Oct 16, 2024
  
  415f76a9
- [Model] Support SDPA attention for Molmo vision backbone (#9410) · cf1d62a6
  Isotr0py authored Oct 16, 2024
  
  cf1d62a6
- [Misc] Consolidate example usage of OpenAI client for multimodal models (#9412) · 59230ef3
  Roger Wang authored Oct 16, 2024
```
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  59230ef3
- [Core] Rename input data types (#8688) · cee711fd
  Cyrus Leung authored Oct 16, 2024
  
  cee711fd
- [CI/Build] Test VLM embeddings (#9406) · 1de76a0e
  Cyrus Leung authored Oct 16, 2024
  
  1de76a0e
- [Model] VLM2Vec, the first multimodal embedding model in vLLM (#9303) · 7abba39e
  Cyrus Leung authored Oct 16, 2024
  
  7abba39e
- [Misc] Standardize RoPE handling for Qwen2-VL (#9250) · 7e7eae33
  Cyrus Leung authored Oct 16, 2024
  
  7e7eae33
- [Bugfix] Molmo text-only input bug fix (#9397) · ed920135
  Reza Salehi authored Oct 15, 2024
```
Co-authored-by: sanghol <sanghol@allenai.org>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
```
  ed920135
- [Bugfix][CI/Build] Fix CUDA 11.8 Build (#9386) · 717a5f82
  Lucas Wilkinson authored Oct 15, 2024
  
  717a5f82
15 Oct, 2024 8 commits
- [Bugfix] Fix vLLM UsageInfo and logprobs None AssertionError with empty token_ids (#9034) · ba309422
  Chang Su authored Oct 15, 2024
```
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
```
  ba309422
- [Misc] Directly use compressed-tensors for checkpoint definitions (#8909) · 22f8a695
  Michael Goin authored Oct 15, 2024
```
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  22f8a695
- pass ignore_eos parameter to all benchmark_serving calls (#9349) · 5d264f4a
  Grace Ho authored Oct 15, 2024
  
  5d264f4a
- [BugFix] Fix chat API continuous usage stats (#9357) · e9d517f2
  Nick Hill authored Oct 15, 2024
  
  e9d517f2
- [Bugfix] Update InternVL input mapper to support image embeds (#9351) · 55e081fb
  hhzhang16 authored Oct 14, 2024
  
  55e081fb
- [Doc] Fix code formatting in spec_decode.rst (#9348) · 8e836d98
  Michael Goin authored Oct 15, 2024
  
  8e836d98
- [Frontend] Clarify model_type error messages (#9345) · 44eaa5a5
  Steve Grubb authored Oct 15, 2024
  
  44eaa5a5
- [Bugfix] Clean up some cruft in mamba.py (#9343) · 169b5306
  Tyler Michael Smith authored Oct 14, 2024
  
  169b5306
14 Oct, 2024 4 commits
- [Model] Make llama3.2 support multiple and interleaved images (#9095) · f0fe4fe8
  Xiang Xu authored Oct 14, 2024
  
  f0fe4fe8
- [Frontend] merge beam search implementations (#9296) · 4d31cd42
  Brendan Wong authored Oct 14, 2024
  
  4d31cd42
- [TPU] Fix TPU SMEM OOM by Pallas paged attention kernel (#9350) · 473e7b36
  Woosuk Kwon authored Oct 14, 2024
  
  473e7b36
- [Docs] Remove PDF build from Readtehdocs (#9347) · fd47e57f
  Simon Mo authored Oct 14, 2024
  
  fd47e57f