Commits · 86e9c8df29a954a7a2fc46e9985fecc2a2e15ae8 · OpenDAS / vllm_cscc · GitLab

23 Sep, 2024 1 commit

[Kernel] (2/N) Machete - Integrate into CompressedTensorsWNA16 and GPTQMarlin (#7701) · 86e9c8df

Lucas Wilkinson authored Sep 23, 2024

Co-authored-by: mgoin <michael@neuralmagic.com>
Co-authored-by: Divakar Verma <137818590+divakar-amd@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>

86e9c8df

21 Sep, 2024 1 commit
- [Core] Rename `PromptInputs` and `inputs`(#8673) · 0057894e
  Cyrus Leung authored Sep 21, 2024
  
  0057894e
19 Sep, 2024 2 commits
- [MISC] remove engine_use_ray in benchmark_throughput.py (#8615) · 855c8ae2
  Kunshang Ji authored Sep 19, 2024
  
  855c8ae2
- [Bugfix] fixing sonnet benchmark bug in benchmark_serving.py (#8616) · c52ec5f0
  Kuntai Du authored Sep 18, 2024
  
  c52ec5f0
18 Sep, 2024 2 commits
- [CI/Build] Update Ruff version (#8469) · 9d104b5b
  Aaron Pham authored Sep 18, 2024
```
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
```
  9d104b5b
- [CI/Build] Avoid CUDA initialization (#8534) · 6ffa3f31
  Cyrus Leung authored Sep 18, 2024
  
  6ffa3f31
17 Sep, 2024 1 commit
- [Benchmark] Support sample from HF datasets and image input for benchmark_serving (#8495) · 1b6de835
  Isotr0py authored Sep 17, 2024
  
  1b6de835
11 Sep, 2024 1 commit
- [Misc] Move device options to a single place (#8322) · 8baa4549
  Aarni Koskela authored Sep 11, 2024
  
  8baa4549
07 Sep, 2024 1 commit
- Enable Random Prefix Caching in Serving Profiling Tool (benchmark_serving.py) (#8241) · 795b662c
  Wei-Sheng Chin authored Sep 06, 2024
  
  795b662c
06 Sep, 2024 1 commit
- [Frontend] Add --logprobs argument to `benchmark_serving.py` (#8191) · e5cab715
  afeldman-nm authored Sep 06, 2024
  
  e5cab715
04 Sep, 2024 2 commits
- [MISC] Replace input token throughput with total token throughput (#8164) · 77d9e514
  Cody Yu authored Sep 04, 2024
```
Co-authored-by: Michael Goin <michael@neuralmagic.com>
```
  77d9e514
- [Benchmark] Add `--async-engine` option to benchmark_throughput.py (#7964) · d4db9f53
  Nick Hill authored Sep 03, 2024
  
  d4db9f53
29 Aug, 2024 1 commit
- Add more percentiles and latencies (#7759) · 0c785d34
  Wei-Sheng Chin authored Aug 29, 2024
  
  0c785d34
27 Aug, 2024 2 commits
- [benchmark] Update TGI version (#7917) · 345be0e2
  Philipp Schmid authored Aug 28, 2024
  
  345be0e2
- [Core] Asynchronous Output Processor (#7049) · 2eedede8
  Megha Agarwal authored Aug 26, 2024
```
Co-authored-by: Alexander Matveev <alexm@neuralmagic.com>
```
  2eedede8
23 Aug, 2024 1 commit
- [Core] Add multi-step support to LLMEngine (#7789) · 9db93de2
  Alexander Matveev authored Aug 23, 2024
  
  9db93de2
22 Aug, 2024 2 commits
- [Misc] Enhance prefix-caching benchmark tool (#6568) · d3b5b980
  Jiaxin Shan authored Aug 23, 2024
  
  d3b5b980
- [Kernel] Replaced `blockReduce[...]` functions with `cub::BlockReduce` (#7233) · 7937009a
  Luka Govedič authored Aug 21, 2024
```
Co-authored-by: Michael Goin <michael@neuralmagic.com>
```
  7937009a
21 Aug, 2024 1 commit
- [misc] Add Torch profiler support (#7451) · dd53c4b0
  William Lin authored Aug 21, 2024
```
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
```
  dd53c4b0
20 Aug, 2024 1 commit
- [Kernel] (1/N) Machete - Hopper Optimized Mixed Precision Linear Kernel (#7174) · 5288c06a
  Lucas Wilkinson authored Aug 20, 2024
  
  5288c06a
16 Aug, 2024 1 commit
- [Kernel] W8A16 Int8 inside FusedMoE (#7415) · 7fc23be8
  Mor Zusman authored Aug 16, 2024
  
  7fc23be8
09 Aug, 2024 1 commit
- [Bugfix] Fix ITL recording in serving benchmark (#7372) · 70d268a3
  Roger Wang authored Aug 09, 2024
  
  70d268a3
06 Aug, 2024 1 commit
- [Kernel] Add per-tensor and per-token AZP epilogues (#5941) · 8d59dbb0
  Luka Govedič authored Aug 06, 2024
```
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
```
  8d59dbb0
02 Aug, 2024 1 commit
- [Misc] Disambiguate quantized types via a new ScalarType (#6396) · a8d604ca
  Lucas Wilkinson authored Aug 02, 2024
  
  a8d604ca
31 Jul, 2024 1 commit
- [Kernel] Tuned int8 Cutlass Kernels for SM75 (T4) (#6996) · 35e9c12b
  Varun Sundar Rabindranath authored Jul 31, 2024
```
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
```
  35e9c12b
29 Jul, 2024 1 commit
- [Kernel] Tuned FP8 Kernels for Ada Lovelace (#6677) · 766435e6
  Varun Sundar Rabindranath authored Jul 29, 2024
```
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
```
  766435e6
27 Jul, 2024 2 commits
- [Kernel] Increase precision of GPTQ/AWQ Marlin kernel (#6795) · 75acdaa4
  Alexander Matveev authored Jul 27, 2024
  
  75acdaa4
- [Model] H2O Danube3-4b (#6451) · 14dbd5a7
  Joe authored Jul 26, 2024
  
  14dbd5a7
22 Jul, 2024 1 commit
- [Frontend] Refactor prompt processing (#4028) · 739b61a3
  Cyrus Leung authored Jul 23, 2024
```
Co-authored-by: Roger Wang <ywang@roblox.com>
```
  739b61a3
17 Jul, 2024 1 commit
- [Misc] Use `torch.Tensor` for type annotation (#6505) · a9a2e74d
  Woosuk Kwon authored Jul 17, 2024
  
  a9a2e74d
16 Jul, 2024 1 commit
- [Kernel][Attention] Separate `Attention.kv_scale` into `k_scale` and `v_scale` (#6081) · 978aed53
  Michael Goin authored Jul 16, 2024
  
  978aed53
15 Jul, 2024 1 commit
- [Bugfix] Benchmark serving script used global parameter 'args' in function... · ccb20db8
  Fish authored Jul 15, 2024
```
[Bugfix] Benchmark serving script used global parameter 'args' in function 'sample_random_requests' (#6428)
```
  ccb20db8
14 Jul, 2024 1 commit
- [Feature] vLLM CLI (#5090) · dbfe254e
  Ethan Xu authored Jul 14, 2024
```
Co-authored-by: simon-mo <simon.mo@hey.com>
```
  dbfe254e
11 Jul, 2024 3 commits
- [CI/Build] Add nightly benchmarking for tgi, tensorrt-llm and lmdeploy (#5362) · a4feba92
  Kuntai Du authored Jul 11, 2024
  
  a4feba92
- [ Misc ] Refactor Marlin Python Utilities (#6082) · b675069d
  Robert Shaw authored Jul 11, 2024
```
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
```
  b675069d
- [Bugfix] Fix snapshot download in serving benchmark (#6318) · c4774eb8
  Roger Wang authored Jul 11, 2024
  
  c4774eb8
08 Jul, 2024 1 commit
- Feature/add benchmark testing (#5947) · 717f4bce
  Haichuan authored Jul 08, 2024
```
Co-authored-by: Roger Wang <ywang@roblox.com>
```
  717f4bce
07 Jul, 2024 1 commit
- add benchmark for fix length input and output (#5857) · 333306a2
  Haichuan authored Jul 07, 2024
```
Co-authored-by: Roger Wang <ywang@roblox.com>
```
  333306a2
02 Jul, 2024 1 commit
- [Core] Optimize block_manager_v2 vs block_manager_v1 (to make V2 default) (#5602) · 3476ed08
  Alexander Matveev authored Jul 01, 2024
  
  3476ed08
01 Jul, 2024 1 commit
- [Frontend] Relax api url assertion for openai benchmarking (#6046) · e373853e
  James Whedbee authored Jul 01, 2024
  
  e373853e