GitLab

[Core] Cross-attention KV caching and memory-management (towards eventual encoder/decoder model support) (#4837)

Co-authored-by: Roger Wang <ywang@roblox.com>

[Core][2/N] Model runner refactoring part 2. Combine prepare prefill / decode to a single API (#4681)

This PR combines prepare_prompt and prepare_decode into a single API. This PR also coelsce the attn metadata for prefill/decode to a single class and allow to slice them when running attn backend.

It also refactors subquery_start_loc which was not refactored in the previous PR

[Core][Hash][Automatic Prefix caching] Accelerating the hashing function by avoiding deep copies (#4696)

Co-authored-by: Cade Daniel <edacih@gmail.com>

Co-authored-by: LiuXiaoxuanPKU <llilyliupku@gmail.com>

Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>

Co-authored-by: Zhong Wang <wangzhong@infini-ai.com>

Co-authored-by: Sahil Suneja <sahilsuneja@gmail.com>

Co-authored-by: Swapnil Parekh <swapnilp@ibm.com>

Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>

Possible fix for conflict between Automated Prefix Caching (#2762) and multi-LoRA support (#1804) (#3263)

Co-authored-by: Avnish Narayan <avnish@anyscale.com>

Co-authored-by: ElizaWszola <eliza@neuralmagic.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>