- 08 Feb, 2025 2 commits
-
-
youkaichao authored
Signed-off-by:youkaichao <youkaichao@gmail.com>
-
Woosuk Kwon authored
Signed-off-by:Woosuk Kwon <woosuk.kwon@berkeley.edu>
-
- 07 Feb, 2025 4 commits
-
-
TJian authored
[ROCm] [Feature] [Doc] [Dockerfile] [BugFix] Support Per-Token-Activation Per-Channel-Weight FP8 Quantization Inferencing (#12501)
-
afeldman-nm authored
This PR is adding support for sample logprobs & prompt logprobs to vLLM v1. New behavior: - During model execution, model runner computes sample logprobs (if user-provided logprobs setting is not None) and prompt logprobs (if user-provided prompt_logprobs setting is not None). For both sample and prompt logprobs, the engine core returns 3 vectors: token ids, token logprob values, token ranks. Ranks reflect tokens' 1-indexed positions in the vocabulary vector after sorting the vocabulary by log probability in descending order. - In scheduler.update_from_output(), sample and prompt logprobs are incorporated into the EngineCoreOutput data structure which is transferred to the engine client. If multiprocessing is enabled, then sample and prompt logprobs will be (de)serialized when the EngineCoreOutput data structure is (de)serialized. - During output processing, the LogprobsProcessor transforms the triplet of token ids, token logprobs values, and token ranks into the OpenAI-compatible List[Dict[token id,Logprob]] format (for sample and prompt logprobs respectively.) - Each Logprob instance (whether sample- or prompt-) consists of a token's log-probability, rank, and detokenized string representation. Note that logprob detokenization is handled by the LogprobsProcessor not the detokenizer. Signed-off-by:
Andrew Feldman <afeldman@neuralmagic.com> Signed-off-by:
Nick Hill <nhill@redhat.com> Signed-off-by:
rshaw@neuralmagic.com <rshaw@neuralmagic.com> Co-authored-by:
rshaw@neuralmagic.com <rshaw@neuralmagic.com> Co-authored-by:
Nick Hill <nhill@redhat.com>
-
Cyrus Leung authored
-
Maximilien de Bayser authored
-
- 06 Feb, 2025 8 commits
-
-
Yu Chin Fabian Lim authored
Signed-off-by:
Yu Chin Fabian Lim <flim@sg.ibm.com> Signed-off-by:
Tyler Michael Smith <tyler@neuralmagic.com> Co-authored-by:
Tyler Michael Smith <tyler@neuralmagic.com>
-
Varun Sundar Rabindranath authored
Signed-off-by:
Varun Sundar Rabindranath <varun@neuralmagic.com> Co-authored-by:
Varun Sundar Rabindranath <varun@neuralmagic.com>
-
youkaichao authored
Signed-off-by:youkaichao <youkaichao@gmail.com>
-
Isotr0py authored
-
Dipika Sikka authored
-
Lu Fang authored
Signed-off-by:Lu Fang <lufang@fb.com>
-
Sumit Vij authored
-
Cyrus Leung authored
-
- 05 Feb, 2025 4 commits
-
-
Roger Wang authored
-
Rahul Tuli authored
-
Lucas Wilkinson authored
Signed-off-by:
simon-mo <xmo@berkeley.edu> Signed-off-by:
Lucas Wilkinson <lcwilkins@redhat.com> Signed-off-by:
Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by:
Lucas Wilkinson <lwilkinson@neuralmagic.com> Co-authored-by:
simon-mo <xmo@berkeley.edu>
-
Mark McLoughlin authored
Signed-off-by:Mark McLoughlin <markmc@redhat.com>
-
- 04 Feb, 2025 7 commits
-
-
Cyrus Leung authored
Signed-off-by:DarkLight1337 <tlleungac@connect.ust.hk>
-
Isotr0py authored
Signed-off-by:
Isotr0py <2037008807@qq.com> Co-authored-by:
Cyrus Leung <cyrus.tl.leung@gmail.com>
-
Woosuk Kwon authored
Signed-off-by:Woosuk Kwon <woosuk.kwon@berkeley.edu>
-
Cyrus Leung authored
Signed-off-by:
DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by:
Isotr0py <2037008807@qq.com> Co-authored-by:
Isotr0py <2037008807@qq.com>
-
Hongxia Yang authored
Signed-off-by:
Hongxia Yang <hongxia.yang@amd.com> Co-authored-by:
Matthew Wong <Matthew.Wong2@amd.com>
-
Thomas Parnell authored
Signed-off-by:Thomas Parnell <tpa@zurich.ibm.com>
-
Russell Bryant authored
Signed-off-by:Russell Bryant <rbryant@redhat.com>
-
- 03 Feb, 2025 5 commits
-
-
Cody Yu authored
Signed-off-by:Cody Yu <hao.yu.cody@gmail.com>
-
Cody Yu authored
Signed-off-by:Cody Yu <hao.yu.cody@gmail.com>
-
Russell Bryant authored
Signed-off-by:Russell Bryant <rbryant@redhat.com>
-
Arthur authored
# Adds support for `transformers` as a backend Following https://github.com/huggingface/transformers/pull/35235 , a bunch of models should already be supported, we are ramping up support for more models. Thanks @Isotr0py for the TP support, and @hmellor for his help as well! This includes: - `trust_remote_code=True` support: any model on the hub, if it implements attention the correct way can be natively supported!! - tensor parallel support --------- Signed-off-by:
Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by:
Isotr0py <2037008807@qq.com> Co-authored-by:
Isotr0py <41363108+Isotr0py@users.noreply.github.com> Co-authored-by:
Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by:
Isotr0py <2037008807@qq.com> Co-authored-by:
Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by:
Michael Goin <mgoin64@gmail.com> Co-authored-by:
Isotr0py <mozf@mail2.sysu.edu.cn>
-
youkaichao authored
When people use deepseek models, they find that they need to solve cv2 version conflict, see https://zhuanlan.zhihu.com/p/21064432691 . I added the check, and make all imports of `cv2` lazy. --------- Signed-off-by:
youkaichao <youkaichao@gmail.com>
-
- 02 Feb, 2025 2 commits
-
-
Russell Bryant authored
- **Add SPDX license headers to python source files** - **Check for SPDX headers using pre-commit** commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745 Author: Russell Bryant <rbryant@redhat.com> Date: Fri Jan 31 14:18:24 2025 -0500 Add SPDX license headers to python source files This commit adds SPDX license headers to python source files as recommended to the project by the Linux Foundation. These headers provide a concise way that is both human and machine readable for communicating license information for each source file. It helps avoid any ambiguity about the license of the code and can also be easily used by tools to help manage license compliance. The Linux Foundation runs license scans against the codebase to help ensure we are in compliance with the licenses of the code we use, including dependencies. Having these headers in place helps that tool do its job. More information can be found on the SPDX site: - https://spdx.dev/learn/handling-license-info/ Signed-off-by:Russell Bryant <rbryant@redhat.com> commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea Author: Russell Bryant <rbryant@redhat.com> Date: Fri Jan 31 14:36:32 2025 -0500 Check for SPDX headers using pre-commit Signed-off-by:
Russell Bryant <rbryant@redhat.com> --------- Signed-off-by:
Russell Bryant <rbryant@redhat.com>
-
Shawn Du authored
As mentioned in RFC https://github.com/vllm-project/vllm/issues/12254 , this PR achieves the task: combine allocate_slots and append_slots. There should be no functionality change, except that in decode, also raise exception when num_tokens is zero (like prefill), and change the unit test case accordingly. @comaniac @rickyyx @WoosukKwon @youkaichao @heheda12345 @simon-mo --------- Signed-off-by:
Shawn Du <shawnd200@outlook.com>
-
- 31 Jan, 2025 3 commits
-
-
Chen Zhang authored
This pr adds extra key to block hash, to generate different hash value for two blocks with the same token string but different extra_keys in their parent blocks. For example, it can generate different hash value for the second block of the following two requests: ```python request1 = make_request( request_id=0, prompt_token_ids=[_ for _ in range(6)], mm_positions=[{ "offset": 0, "length": 3 }, { "offset": 3, "length": 3 }], mm_hashes=["hash1", "hash2"], ) request2 = make_request( request_id=1, prompt_token_ids=[_ for _ in range(6)], mm_positions=[{ "offset": 0, "length": 3 }, { "offset": 3, "length": 3 }], mm_hashes=["hash3", "hash2"], ) ``` --------- Signed-off-by:Chen Zhang <zhangch99@outlook.com>
-
Lucas Wilkinson authored
Signed-off-by:
Lucas Wilkinson <lwilkinson@neuralmagic.com> Signed-off-by:
simon-mo <xmo@berkeley.edu> Co-authored-by:
Woosuk Kwon <woosuk.kwon@berkeley.edu> Co-authored-by:
simon-mo <simon.mo@hey.com> Co-authored-by:
Michael Goin <mgoin64@gmail.com> Co-authored-by:
Zhuohan Li <zhuohan123@gmail.com> Co-authored-by:
Tyler Michael Smith <tysmith@redhat.com> Co-authored-by:
Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com> Co-authored-by:
simon-mo <xmo@berkeley.edu>
-
Lucas Wilkinson authored
-
- 30 Jan, 2025 1 commit
-
-
Mark McLoughlin authored
Signed-off-by:Mark McLoughlin <markmc@redhat.com>
-
- 29 Jan, 2025 4 commits
-
-
Jinzhen Lin authored
-
Yanyi Liu authored
Signed-off-by:liuyanyi <wolfsonliu@163.com>
-
Alphi authored
Signed-off-by:
hzh <hezhihui_thu@163.com> Signed-off-by:
Sungjae Lee <33976427+llsj14@users.noreply.github.com> Signed-off-by:
shaochangxu.scx <shaochangxu.scx@antgroup.com> Signed-off-by:
DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by:
NickLucche <nlucches@redhat.com> Signed-off-by:
Isotr0py <2037008807@qq.com> Signed-off-by:
Roger Wang <ywang@roblox.com> Signed-off-by:
Rafael Vasquez <rafvasq21@gmail.com> Signed-off-by:
Akshat Tripathi <akshat@krai.ai> Signed-off-by:
Oleg Mosalov <oleg@krai.ai> Signed-off-by:
Jee Jee Li <pandaleefree@gmail.com> Signed-off-by:
rshaw@neuralmagic.com <rshaw@neuralmagic.com> Signed-off-by:
Yida Wu <yidawu@alumni.cmu.edu> Signed-off-by:
Chenguang Li <757486878@qq.com> Signed-off-by:
youkaichao <youkaichao@gmail.com> Signed-off-by:
Alex-Brooks <Alex.brooks@ibm.com> Signed-off-by:
Chen Zhang <zhangch99@outlook.com> Signed-off-by:
Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by:
Shanshan Shen <467638484@qq.com> Signed-off-by:
elijah <f1renze.142857@gmail.com> Signed-off-by:
Yikun <yikunkero@gmail.com> Signed-off-by:
mgoin <michael@neuralmagic.com> Signed-off-by:
Woosuk Kwon <woosuk.kwon@berkeley.edu> Signed-off-by:
Konrad Zawora <kzawora@habana.ai> Signed-off-by:
tjtanaa <tunjian.tan@embeddedllm.com> Signed-off-by:
wangxiyuan <wangxiyuan1007@gmail.com> Signed-off-by:
Rui Qiao <ruisearch42@gmail.com> Co-authored-by:
Sungjae Lee <33976427+llsj14@users.noreply.github.com> Co-authored-by:
shaochangxu <85155497+shaochangxu@users.noreply.github.com> Co-authored-by:
shaochangxu.scx <shaochangxu.scx@antgroup.com> Co-authored-by:
Cyrus Leung <tlleungac@connect.ust.hk> Co-authored-by:
Nicolò Lucchesi <nlucches@redhat.com> Co-authored-by:
sixgod <evethwillbeok@outlook.com> Co-authored-by:
Isotr0py <2037008807@qq.com> Co-authored-by:
Roger Wang <136131678+ywang96@users.noreply.github.com> Co-authored-by:
Rafael Vasquez <rafvasq21@gmail.com> Co-authored-by:
Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by:
Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by:
Akshat Tripathi <Akshat.tripathi6568@gmail.com> Co-authored-by:
Oleg Mosalov <oleg@krai.ai> Co-authored-by:
Jee Jee Li <pandaleefree@gmail.com> Co-authored-by:
Avshalom Manevich <12231371+avshalomman@users.noreply.github.com> Co-authored-by:
Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com> Co-authored-by:
Yangcheng Li <liyangcheng.lyc@alibaba-inc.com> Co-authored-by:
Siyuan Li <94890248+liaoyanqing666@users.noreply.github.com> Co-authored-by:
Concurrensee <yida.wu@amd.com> Co-authored-by:
Chenguang Li <757486878@qq.com> Co-authored-by:
youkaichao <youkaichao@gmail.com> Co-authored-by:
Alex Brooks <alex.brooks@ibm.com> Co-authored-by:
Chen Zhang <zhangch99@outlook.com> Co-authored-by:
Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by:
Shanshan Shen <467638484@qq.com> Co-authored-by:
elijah <30852919+e1ijah1@users.noreply.github.com> Co-authored-by:
Yikun Jiang <yikunkero@gmail.com> Co-authored-by:
Steve Luo <36296769+SunflowerAries@users.noreply.github.com> Co-authored-by:
mgoin <michael@neuralmagic.com> Co-authored-by:
Woosuk Kwon <woosuk.kwon@berkeley.edu> Co-authored-by:
Konrad Zawora <kzawora@habana.ai> Co-authored-by:
TJian <tunjian1996@gmail.com> Co-authored-by:
tjtanaa <tunjian.tan@embeddedllm.com> Co-authored-by:
wangxiyuan <wangxiyuan1007@gmail.com> Co-authored-by:
maang-h <55082429+maang-h@users.noreply.github.com> Co-authored-by:
Elfie Guo <164945471+elfiegg@users.noreply.github.com> Co-authored-by:
Rui Qiao <161574667+ruisearch42@users.noreply.github.com> Co-authored-by:
Roger Wang <ywang@roblox.com>
-
Travis Johnson authored
Signed-off-by:
Travis Johnson <tsjohnso@us.ibm.com> Signed-off-by:
Wallas Santos <wallashss@ibm.com> Co-authored-by:
Wallas Santos <wallashss@ibm.com>
-