GitLab

Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>

Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>

Co-authored-by: Breno Faria <breno.faria@intrafind.com>

[Core] Cross-attention KV caching and memory-management (towards eventual encoder/decoder model support) (#4837)

Co-authored-by: Roger Wang <ywang@roblox.com>

Co-authored-by: Ruth Evans <ruthevans@Ruths-MacBook-Pro.local>

Co-authored-by: rsnm2 <rshaw@neuralmagic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>

Co-authored-by: beagleski <yunanzhang@microsoft.com>
Co-authored-by: bapatra <bapatra@microsoft.com>
Co-authored-by: Barun Patra <codedecde@users.noreply.github.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>

Co-authored-by: Lei Wen <wenlei03@qiyi.com>

Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>

Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>

Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>

The 2nd PR for #4532.

This PR supports loading FP8 kv-cache scaling factors from a FP8 checkpoint (with .kv_scale parameter).

Pass the CUDA stream into the CUTLASS GEMMs, to avoid future issues with CUDA graphs

Co-authored-by: Alexey Kondratiev <alexey.kondratiev@amd.com>

Currently we need to call rotary embedding kernel for each LoRA, which makes it hard to serve multiple long context length LoRA. Add batched rotary embedding kernel and pipe it through.

It replaces the rotary embedding layer to the one that is aware of multiple cos-sin-cache per scaling factors.

Follow up of https://github.com/vllm-project/vllm/pull/3095/files

[Build/CI] Extending the set of AMD tests with Regression, Basic Correctness, Distributed, Engine, Llava Tests (#4797)

Co-authored-by: Silencio <silencio@adsl-99-6-187-6.dsl.irvnca.sbcglobal.net>