GitLab

Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>

Co-authored-by: ywang96 <ywang@roblox.com>

 [Speculative Decoding] Support draft model on different tensor-parallel size than target model (#5414)

This PR adds basic support for OpenTelemetry distributed tracing.
It includes changes to enable tracing functionality and improve monitoring capabilities.

I've also added a markdown with print-screens to guide users how to use this feature. You can find it here

Co-authored-by: Jiang Li <jiang1.li@intel.com>
Co-authored-by: Abhilash Majumder <abhilash.majumder@intel.com>
Co-authored-by: Abhilash Majumder <30946547+abhilash1910@users.noreply.github.com>

Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>

Co-authored-by: Roger Wang <ywang@roblox.com>

Co-authored-by: Ruth Evans <ruthevans@Ruths-MacBook-Pro.local>

Co-authored-by: rsnm2 <rshaw@neuralmagic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>

The 2nd PR for #4532.

This PR supports loading FP8 kv-cache scaling factors from a FP8 checkpoint (with .kv_scale parameter).

Signed-off-by: kerthcet <kerthcet@gmail.com>

Currently we need to call rotary embedding kernel for each LoRA, which makes it hard to serve multiple long context length LoRA. Add batched rotary embedding kernel and pipe it through.

It replaces the rotary embedding layer to the one that is aware of multiple cos-sin-cache per scaling factors.

Follow up of https://github.com/vllm-project/vllm/pull/3095/files