Commits · ce4f5a29fb3e35041842518fefe999847b8326b9 · OpenDAS / vllm_cscc

02 Mar, 2024 1 commit

Add Automatic Prefix Caching (#2762) · ce4f5a29

Sage Moore authored Mar 02, 2024


Co-authored-by: ElizaWszola <eliza@neuralmagic.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>

ce4f5a29

01 Mar, 2024 1 commit
- Fix: Output text is always truncated in some models (#3016) · 54d35447
  Sherry authored Mar 01, 2024
  
  54d35447
29 Feb, 2024 2 commits
- [Fix] Don't deep-copy LogitsProcessors when copying SamplingParams (#3099) · 29a8d6a5
  Nick Hill authored Feb 29, 2024
  
  29a8d6a5
- add cache_config's info to prometheus metrics. (#3100) · 9289e577
  Allen.Dou authored Feb 29, 2024
  
  9289e577
28 Feb, 2024 1 commit
- [Neuron] Support inference with transformers-neuronx (#2569) · 3b7178cf
  Liangfu Chen authored Feb 28, 2024
  
  3b7178cf
27 Feb, 2024 1 commit
- Don't use cupy when `enforce_eager=True` (#3037) · c1c0d00b
  Roy authored Feb 27, 2024
  
  c1c0d00b
25 Feb, 2024 1 commit
- Port metrics from `aioprometheus` to `prometheus_client` (#2730) · ef978fe4
  Harry Mellor authored Feb 25, 2024
  
  ef978fe4
22 Feb, 2024 1 commit
- Include tokens from prompt phase in `counter_generation_tokens` (#2802) · 4caf7044
  Ronen Schaffer authored Feb 23, 2024
  
  4caf7044
21 Feb, 2024 1 commit
- Add metrics to RequestOutput (#2876) · 017d9f15
  Antoni Baum authored Feb 20, 2024
  
  017d9f15
19 Feb, 2024 1 commit
- Fix `vllm:prompt_tokens_total` metric calculation (#2869) · e433c115
  Ronen Schaffer authored Feb 19, 2024
  
  e433c115
17 Feb, 2024 1 commit

Defensively copy `sampling_params` (#2881) · 185b2c29

Nick Hill authored Feb 17, 2024

If the SamplingParams object passed to LLMEngine.add_request() is mutated after it returns, it could affect the async sampling process for that request.

Suggested by @Yard1 https://github.com/vllm-project/vllm/pull/2514#discussion_r1490106059

185b2c29

13 Feb, 2024 1 commit
- Use CuPy for CUDA graphs (#2811) · a463c333
  Woosuk Kwon authored Feb 13, 2024
  
  a463c333
08 Feb, 2024 1 commit
- [Ray] Integration compiled DAG off by default (#2471) · 65b89d16
  SangBin Cho authored Feb 09, 2024
  
  65b89d16
04 Feb, 2024 1 commit
- Remove eos tokens from output by default (#2611) · 5a6c81b0
  Rex authored Feb 04, 2024
  
  5a6c81b0
01 Feb, 2024 1 commit
- Remove hardcoded `device="cuda" ` to support more devices (#2503) · 96b6f475
  Kunshang Ji authored Feb 02, 2024
```
Co-authored-by: Jiang Li <jiang1.li@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
```
  96b6f475
31 Jan, 2024 1 commit
- Refactor Prometheus and Add Request Level Metrics (#2316) · 93b38bea
  Robert Shaw authored Jan 31, 2024
  
  93b38bea
29 Jan, 2024 2 commits
- Fix error when tp > 1 (#2644) · b72af8f1
  zhaoyang-star authored Jan 29, 2024
```
Co-authored-by: zhaoyang-star <zhao.yang16@zte.com.cn>
```
  b72af8f1
- Support FP8-E5M2 KV Cache (#2279) · 9090bf02
  zhaoyang-star authored Jan 29, 2024
```
Co-authored-by: zhaoyang <zhao.yang16@zte.com.cn>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
```
  9090bf02
27 Jan, 2024 1 commit
- Implement custom all reduce kernels (#2192) · 38017003
  Hanzhi Zhou authored Jan 28, 2024
  
  38017003
23 Jan, 2024 1 commit

[Experimental] Add multi-LoRA support (#1804) · 9b945daa

Antoni Baum authored Jan 24, 2024


Co-authored-by: Chen Shen <scv119@gmail.com>
Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com>
Co-authored-by: Avnish Narayan <avnish@anyscale.com>

9b945daa

22 Jan, 2024 2 commits
- Fix https://github.com/vllm-project/vllm/issues/2540 (#2545) · ab7e6006
  Philipp Moritz authored Jan 22, 2024
  
  ab7e6006
- [Speculative decoding 2/9] Multi-step worker for draft model (#2424) · 18bfcdd0
  Cade Daniel authored Jan 21, 2024
  
  18bfcdd0
18 Jan, 2024 1 commit

[Experimental] Prefix Caching Support (#1669) · d10f8e1d

shiyi.c_98 authored Jan 17, 2024


Co-authored-by: DouHappy <2278958187@qq.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>

d10f8e1d

12 Jan, 2024 1 commit
- [DOC] Add additional comments for LLMEngine and AsyncLLMEngine (#1011) · 6549aef2
  Jiaxiang authored Jan 12, 2024
  
  6549aef2
07 Jan, 2024 1 commit
- Changed scheduler to use deques instead of lists (#2290) · 05921a9a
  Nadav Shmayovits authored Jan 07, 2024
```
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  05921a9a
05 Jan, 2024 1 commit
- Ensure metrics are logged regardless of requests (#2347) · d0215a58
  Iskren Ivov Chernev authored Jan 05, 2024
  
  d0215a58
03 Jan, 2024 1 commit
- Use NCCL instead of ray for control-plane communication to remove serialization overhead (#2221) · fd4ea8ef
  Zhuohan Li authored Jan 04, 2024
  
  fd4ea8ef
26 Dec, 2023 1 commit
- [BUGFIX] Do not return ignored sentences twice in async llm engine (#2258) · e0ff9200
  Zhuohan Li authored Dec 26, 2023
  
  e0ff9200
21 Dec, 2023 1 commit
- Disable Ray usage stats collection (#2206) · 3a4fd5ca
  Woosuk Kwon authored Dec 20, 2023
  
  3a4fd5ca
18 Dec, 2023 1 commit
- [BugFix] Raise error when max_model_len is larger than KV cache (#2163) · 8041b730
  Woosuk Kwon authored Dec 17, 2023
  
  8041b730
17 Dec, 2023 2 commits
- Remove dependency on CuPy (#2152) · c3372e87
  Woosuk Kwon authored Dec 17, 2023
  
  c3372e87
- Optimize model execution with CUDA graph (#1926) · 37ca5581
  Woosuk Kwon authored Dec 16, 2023
```
Co-authored-by: Chen Shen <scv119@gmail.com>
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
```
  37ca5581
15 Dec, 2023 1 commit
- Add a flag to include stop string in output text (#1976) · c06170cc
  Yunfeng Bai authored Dec 15, 2023
  
  c06170cc
03 Dec, 2023 2 commits
- Fix num_gpus when TP > 1 (#1852) · 464dd985
  Woosuk Kwon authored Dec 03, 2023
  
  464dd985
- Add Production Metrics in Prometheus format (#1890) · 5313c2cb
  Simon Mo authored Dec 02, 2023
  
  5313c2cb
30 Nov, 2023 1 commit
- Refactor Worker & InputMetadata (#1843) · 27feead2
  Woosuk Kwon authored Nov 29, 2023
  
  27feead2
29 Nov, 2023 1 commit
- Better integration with Ray Serve (#1821) · 0229c386
  FlorianJoncour authored Nov 29, 2023
```
Co-authored-by: FlorianJoncour <florian@zetta-sys.com>
```
  0229c386
28 Nov, 2023 1 commit
- [FIX] Fix class naming (#1803) · 708e6c18
  Zhuohan Li authored Nov 28, 2023
  
  708e6c18
21 Nov, 2023 1 commit
- fix RAM OOM when load large models in tensor parallel mode. (#1395) · 4bb6b671
  boydfd authored Nov 21, 2023
```
Co-authored-by: ran_lin <rlin@thoughtworks.com>
```
  4bb6b671
20 Nov, 2023 1 commit
- Migrate linter from `pylint` to `ruff` (#1665) · 5ffc0d13
  Simon Mo authored Nov 20, 2023
  
  5ffc0d13