Commits · d69ff0cbbbb07b571eeea62b4e2ce87b91cea387 · kecinstone / 2024pra-vllm

31 Jan, 2024 1 commit
- Fixes assertion failure in prefix caching: the lora index mapping should respect prefix_len (#2688) · d69ff0cb
  Tao He authored Feb 01, 2024
```
Signed-off-by: Tao He <sighingnow@gmail.com>
```
  d69ff0cb
29 Jan, 2024 1 commit

Support FP8-E5M2 KV Cache (#2279) · 9090bf02

zhaoyang-star authored Jan 29, 2024


Co-authored-by: zhaoyang <zhao.yang16@zte.com.cn>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>

9090bf02

27 Jan, 2024 1 commit
- Implement custom all reduce kernels (#2192) · 38017003
  Hanzhi Zhou authored Jan 28, 2024
  
  38017003
23 Jan, 2024 1 commit

[Experimental] Add multi-LoRA support (#1804) · 9b945daa

Antoni Baum authored Jan 24, 2024


Co-authored-by: Chen Shen <scv119@gmail.com>
Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com>
Co-authored-by: Avnish Narayan <avnish@anyscale.com>

9b945daa

22 Jan, 2024 1 commit
- [Speculative decoding 2/9] Multi-step worker for draft model (#2424) · 18bfcdd0
  Cade Daniel authored Jan 21, 2024
  
  18bfcdd0
19 Jan, 2024 1 commit
- Simplify broadcast logic for control messages (#2501) · ef9b636e
  Zhuohan Li authored Jan 19, 2024
  
  ef9b636e
18 Jan, 2024 1 commit

[Experimental] Prefix Caching Support (#1669) · d10f8e1d

shiyi.c_98 authored Jan 17, 2024


Co-authored-by: DouHappy <2278958187@qq.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>

d10f8e1d

14 Jan, 2024 1 commit
- [Minor] Optimize cuda graph memory usage (#2437) · 9f659bf0
  Roy authored Jan 15, 2024
  
  9f659bf0
12 Jan, 2024 3 commits
- [Minor] Fix err msg (#2431) · 35c4bc20
  Woosuk Kwon authored Jan 12, 2024
  
  35c4bc20
- Suggest using dtype=half when OOM. · cb7a1c1c
  Ben authored Jan 13, 2024
  
  cb7a1c1c
- [DOC] Add additional comments for LLMEngine and AsyncLLMEngine (#1011) · 6549aef2
  Jiaxiang authored Jan 12, 2024
  
  6549aef2
08 Jan, 2024 1 commit
- Fix eager mode performance (#2377) · c8848191
  Woosuk Kwon authored Jan 08, 2024
  
  c8848191
03 Jan, 2024 1 commit
- Use NCCL instead of ray for control-plane communication to remove serialization overhead (#2221) · fd4ea8ef
  Zhuohan Li authored Jan 04, 2024
  
  fd4ea8ef
20 Dec, 2023 1 commit
- Make _prepare_sample non-blocking and use pinned memory for input buffers (#2207) · 31bff691
  Hanzhi Zhou authored Dec 19, 2023
  
  31bff691
19 Dec, 2023 1 commit
- Add warning on CUDA graph memory usage (#2182) · 21d5daa4
  Woosuk Kwon authored Dec 18, 2023
  
  21d5daa4
17 Dec, 2023 3 commits
- Remove dependency on CuPy (#2152) · c3372e87
  Woosuk Kwon authored Dec 17, 2023
  
  c3372e87
- Fix all-reduce memory usage (#2151) · e1d54022
  Woosuk Kwon authored Dec 17, 2023
  
  e1d54022
- Optimize model execution with CUDA graph (#1926) · 37ca5581
  Woosuk Kwon authored Dec 16, 2023
```
Co-authored-by: Chen Shen <scv119@gmail.com>
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
```
  37ca5581
13 Dec, 2023 2 commits
- [BugFix] Fix input positions for long context with sliding window (#2088) · f1c85201
  Woosuk Kwon authored Dec 13, 2023
  
  f1c85201
- Fix peak memory profiling (#2031) · 30bad5c4
  Woosuk Kwon authored Dec 12, 2023
  
  30bad5c4
03 Dec, 2023 1 commit
- Fix broken sampler tests (#1896) · 5f09cbdb
  Woosuk Kwon authored Dec 02, 2023
```
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
```
  5f09cbdb
30 Nov, 2023 1 commit
- Refactor Worker & InputMetadata (#1843) · 27feead2
  Woosuk Kwon authored Nov 29, 2023
  
  27feead2
24 Nov, 2023 1 commit
- [Build] Avoid building too many extensions (#1624) · e0c6f556
  Yanming W authored Nov 23, 2023
  
  e0c6f556
21 Nov, 2023 1 commit
- fix RAM OOM when load large models in tensor parallel mode. (#1395) · 4bb6b671
  boydfd authored Nov 21, 2023
```
Co-authored-by: ran_lin <rlin@thoughtworks.com>
```
  4bb6b671
20 Nov, 2023 1 commit
- Migrate linter from `pylint` to `ruff` (#1665) · 5ffc0d13
  Simon Mo authored Nov 20, 2023
  
  5ffc0d13
08 Nov, 2023 1 commit
- Fix input_metadata.selected_token_indices in worker prepare_inputs (#1546) · 8efe23f1
  Yanming W authored Nov 09, 2023
  
  8efe23f1
01 Nov, 2023 1 commit
- Force paged attention v2 for long contexts (#1510) · 9738b84a
  Antoni Baum authored Nov 01, 2023
  
  9738b84a
31 Oct, 2023 1 commit
- Fix integer overflows in attention & cache ops (#1514) · 0ce8647d
  Woosuk Kwon authored Oct 31, 2023
  
  0ce8647d
30 Oct, 2023 1 commit
- Delay GPU->CPU sync in sampling (#1337) · 15f56323
  Antoni Baum authored Oct 30, 2023
  
  15f56323
17 Oct, 2023 1 commit
- Change scheduler & input tensor shape (#1381) · c1376e0f
  Woosuk Kwon authored Oct 16, 2023
  
  c1376e0f
08 Oct, 2023 1 commit
- Move bfloat16 check to worker (#1259) · ee92b58b
  Antoni Baum authored Oct 07, 2023
  
  ee92b58b
28 Sep, 2023 3 commits
- [Fix] Remove false assertion (#1222) · 2e8e49fc
  Woosuk Kwon authored Sep 28, 2023
  
  2e8e49fc
- Fix Mistral model (#1220) · a8e98aee
  Woosuk Kwon authored Sep 28, 2023
  
  a8e98aee
- [Mistral] Mistral-7B-v0.1 support (#1196) · bb1ba58f
  Chris Bamford authored Sep 28, 2023
```
Co-authored-by: timlacroix <t@mistral.ai>
```
  bb1ba58f
27 Sep, 2023 1 commit
- Allocate more shared memory to attention kernel (#1154) · cf5cb1e3
  Antoni Baum authored Sep 26, 2023
  
  cf5cb1e3
24 Sep, 2023 1 commit
- Fix config for Falcon (#1164) · 9f6be869
  Woosuk Kwon authored Sep 23, 2023
  
  9f6be869
21 Sep, 2023 1 commit
- Replace DtypeTensor (#1123) · 2ac4d5e2
  Woosuk Kwon authored Sep 21, 2023
  
  2ac4d5e2
05 Sep, 2023 1 commit
- Align vLLM's beam search implementation with HF generate (#857) · 002800f0
  Zhuohan Li authored Sep 04, 2023
  
  002800f0
02 Aug, 2023 1 commit
- Add Falcon support (new) (#592) · 1b0bd0fe
  Zhuohan Li authored Aug 02, 2023
  
  1b0bd0fe
20 Jul, 2023 1 commit
- Ray placement group support (#397) · 9925c179
  Antoni Baum authored Jul 19, 2023
  
  9925c179