Commits · 168cab6bbfb733f97defc8c1aa13df90c5319f19 · OpenDAS / vllm_cscc

06 Oct, 2024 1 commit
- [Frontend] API support for beam search (#9087) · 168cab6b
  Brendan Wong authored Oct 05, 2024
```
Co-authored-by: youkaichao <youkaichao@126.com>
```
  168cab6b
04 Oct, 2024 3 commits
- [Misc] Fix CI lint (#9085) · 27302dd5
  Cody Yu authored Oct 04, 2024
  
  27302dd5
- [Misc] Add random seed for prefix cache benchmark (#9081) · 0cc566ca
  Andy Dai authored Oct 04, 2024
  
  0cc566ca
- [CI] Update performance benchmark: upgrade trt-llm to r24.07, and add SGLang (#7412) · fbb74420
  Kuntai Du authored Oct 04, 2024
  
  fbb74420
01 Oct, 2024 1 commit

Update benchmark_serving.py to read and write json-datasets, results in UTF8,... · 22f5851b

vlsav authored Oct 01, 2024

Update benchmark_serving.py to read and write json-datasets, results in UTF8, for better compatibility with Windows (#8997)

22f5851b

28 Sep, 2024 1 commit
- [Bugfix] Support testing prefill throughput with benchmark_serving.py --hf-output-len 1 (#8891) · e585b583
  Chen Zhang authored Sep 28, 2024
  
  e585b583
27 Sep, 2024 2 commits
- [MISC] Fix invalid escape sequence '\' (#8830) · 0e088750
  Peter Pan authored Sep 27, 2024
```
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
```
  0e088750
- [Core] rename`PromptInputs` and `inputs` (#8876) · 3b00b9c2
  Cyrus Leung authored Sep 27, 2024
  
  3b00b9c2
25 Sep, 2024 3 commits
- Revert "rename PromptInputs and inputs with backward compatibility (#8760) (#8810) · 4f1ba084
  Simon Mo authored Sep 25, 2024
  
  4f1ba084
- rename PromptInputs and inputs with backward compatibility (#8760) · 28e1299e
  Cyrus Leung authored Sep 26, 2024
  
  28e1299e
- [Core] Adding Priority Scheduling (#5958) · 6da1ab6b
  Archit Patke authored Sep 24, 2024
  
  6da1ab6b
24 Sep, 2024 2 commits
- Revert "[Core] Rename `PromptInputs` to `PromptType`, and `inputs` to `prompt`" (#8750) · 3185fb0c
  Simon Mo authored Sep 23, 2024
  
  3185fb0c
- re-implement beam search on top of vllm core (#8726) · 0250dd68
  youkaichao authored Sep 23, 2024
```
Co-authored-by: Brendan Wong <bjwpokemon@gmail.com>
```
  0250dd68
23 Sep, 2024 1 commit

[Kernel] (2/N) Machete - Integrate into CompressedTensorsWNA16 and GPTQMarlin (#7701) · 86e9c8df

Lucas Wilkinson authored Sep 23, 2024

Co-authored-by: mgoin <michael@neuralmagic.com>
Co-authored-by: Divakar Verma <137818590+divakar-amd@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>

86e9c8df

21 Sep, 2024 1 commit
- [Core] Rename `PromptInputs` and `inputs`(#8673) · 0057894e
  Cyrus Leung authored Sep 21, 2024
  
  0057894e
19 Sep, 2024 2 commits
- [MISC] remove engine_use_ray in benchmark_throughput.py (#8615) · 855c8ae2
  Kunshang Ji authored Sep 19, 2024
  
  855c8ae2
- [Bugfix] fixing sonnet benchmark bug in benchmark_serving.py (#8616) · c52ec5f0
  Kuntai Du authored Sep 18, 2024
  
  c52ec5f0
18 Sep, 2024 2 commits
- [CI/Build] Update Ruff version (#8469) · 9d104b5b
  Aaron Pham authored Sep 18, 2024
```
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
```
  9d104b5b
- [CI/Build] Avoid CUDA initialization (#8534) · 6ffa3f31
  Cyrus Leung authored Sep 18, 2024
  
  6ffa3f31
17 Sep, 2024 1 commit
- [Benchmark] Support sample from HF datasets and image input for benchmark_serving (#8495) · 1b6de835
  Isotr0py authored Sep 17, 2024
  
  1b6de835
11 Sep, 2024 1 commit
- [Misc] Move device options to a single place (#8322) · 8baa4549
  Aarni Koskela authored Sep 11, 2024
  
  8baa4549
07 Sep, 2024 1 commit
- Enable Random Prefix Caching in Serving Profiling Tool (benchmark_serving.py) (#8241) · 795b662c
  Wei-Sheng Chin authored Sep 06, 2024
  
  795b662c
06 Sep, 2024 1 commit
- [Frontend] Add --logprobs argument to `benchmark_serving.py` (#8191) · e5cab715
  afeldman-nm authored Sep 06, 2024
  
  e5cab715
04 Sep, 2024 2 commits
- [MISC] Replace input token throughput with total token throughput (#8164) · 77d9e514
  Cody Yu authored Sep 04, 2024
```
Co-authored-by: Michael Goin <michael@neuralmagic.com>
```
  77d9e514
- [Benchmark] Add `--async-engine` option to benchmark_throughput.py (#7964) · d4db9f53
  Nick Hill authored Sep 03, 2024
  
  d4db9f53
29 Aug, 2024 1 commit
- Add more percentiles and latencies (#7759) · 0c785d34
  Wei-Sheng Chin authored Aug 29, 2024
  
  0c785d34
27 Aug, 2024 2 commits
- [benchmark] Update TGI version (#7917) · 345be0e2
  Philipp Schmid authored Aug 28, 2024
  
  345be0e2
- [Core] Asynchronous Output Processor (#7049) · 2eedede8
  Megha Agarwal authored Aug 26, 2024
```
Co-authored-by: Alexander Matveev <alexm@neuralmagic.com>
```
  2eedede8
23 Aug, 2024 1 commit
- [Core] Add multi-step support to LLMEngine (#7789) · 9db93de2
  Alexander Matveev authored Aug 23, 2024
  
  9db93de2
22 Aug, 2024 2 commits
- [Misc] Enhance prefix-caching benchmark tool (#6568) · d3b5b980
  Jiaxin Shan authored Aug 23, 2024
  
  d3b5b980
- [Kernel] Replaced `blockReduce[...]` functions with `cub::BlockReduce` (#7233) · 7937009a
  Luka Govedič authored Aug 21, 2024
```
Co-authored-by: Michael Goin <michael@neuralmagic.com>
```
  7937009a
21 Aug, 2024 1 commit
- [misc] Add Torch profiler support (#7451) · dd53c4b0
  William Lin authored Aug 21, 2024
```
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
```
  dd53c4b0
20 Aug, 2024 1 commit
- [Kernel] (1/N) Machete - Hopper Optimized Mixed Precision Linear Kernel (#7174) · 5288c06a
  Lucas Wilkinson authored Aug 20, 2024
  
  5288c06a
16 Aug, 2024 1 commit
- [Kernel] W8A16 Int8 inside FusedMoE (#7415) · 7fc23be8
  Mor Zusman authored Aug 16, 2024
  
  7fc23be8
09 Aug, 2024 1 commit
- [Bugfix] Fix ITL recording in serving benchmark (#7372) · 70d268a3
  Roger Wang authored Aug 09, 2024
  
  70d268a3
06 Aug, 2024 1 commit
- [Kernel] Add per-tensor and per-token AZP epilogues (#5941) · 8d59dbb0
  Luka Govedič authored Aug 06, 2024
```
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
```
  8d59dbb0
02 Aug, 2024 1 commit
- [Misc] Disambiguate quantized types via a new ScalarType (#6396) · a8d604ca
  Lucas Wilkinson authored Aug 02, 2024
  
  a8d604ca
31 Jul, 2024 1 commit
- [Kernel] Tuned int8 Cutlass Kernels for SM75 (T4) (#6996) · 35e9c12b
  Varun Sundar Rabindranath authored Jul 31, 2024
```
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
```
  35e9c12b
29 Jul, 2024 1 commit
- [Kernel] Tuned FP8 Kernels for Ada Lovelace (#6677) · 766435e6
  Varun Sundar Rabindranath authored Jul 29, 2024
```
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
```
  766435e6
27 Jul, 2024 1 commit
- [Kernel] Increase precision of GPTQ/AWQ Marlin kernel (#6795) · 75acdaa4
  Alexander Matveev authored Jul 27, 2024
  
  75acdaa4