GitLab

Signed-off-by: luka <luka@neuralmagic.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>

Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Co-authored-by: Aaron Pham <contact@aarnphm.xyz>

This PR provides initial support for single-node disaggregated prefill in 1P1D scenario.
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
Co-authored-by: ApostaC <yihua98@uchicago.edu>
Co-authored-by: YaoJiayi <120040070@link.cuhk.edu.cn>

Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Isotr0py <2037008807@qq.com>

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Signed-off-by: rickyx <rickyx@anyscale.com>

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>

Signed-off-by: rbbang <anjaehyun87@gmail.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: Cody Yu <hao.yu.cody@gmail.com>

Signed-off-by: Russell Bryant <rbryant@redhat.com>

Signed-off-by: Mozhou <spli161006@gmail.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

Signed-off-by: Linkun Chen <github+anyscale@lkchen.net>
Co-authored-by: Linkun Chen <github+anyscale@lkchen.net>

Signed-off-by: Linkun Chen <github+anyscale@lkchen.net>
Co-authored-by: Linkun Chen <lkchen@github.com>
Co-authored-by: Linkun Chen <github+anyscale@lkchen.net>

Signed-off-by: daitran2k1 <tranquangdai7a@gmail.com>

[Misc][OpenAI] deprecate max_tokens in favor of new max_completion_tokens field for chat completion endpoint (#9837)

Signed-off-by: wangshuai09 <391746016@qq.com>

Signed-off-by: youkaichao <youkaichao@gmail.com>

Signed-off-by: Russell Bryant <rbryant@redhat.com>

Co-authored-by: Isotr0py <2037008807@qq.com>

Removing the block manager v1. This is the initial piece of prefix-caching-centric design. In order to achieve prefix-caching-centric design, we need to simplify the code path so that we only use v2 block manager (which has much higher performance on prefix caching).

[Core] Add an environment variable which needs to be set explicitly to allow BlockSpaceManagerV1 (#9149)

Co-authored-by: youkaichao <youkaichao@126.com>

Update benchmark_serving.py to read and write json-datasets, results in UTF8, for better compatibility with Windows (#8997)