GitLab

[Speculative Decoding] Enabling bonus token in speculative decoding for KV cache based models (#5765)

 [Speculative Decoding] Support draft model on different tensor-parallel size than target model (#5414)

[Core][2/N] Model runner refactoring part 2. Combine prepare prefill / decode to a single API (#4681)

This PR combines prepare_prompt and prepare_decode into a single API. This PR also coelsce the attn metadata for prefill/decode to a single class and allow to slice them when running attn backend.

It also refactors subquery_start_loc which was not refactored in the previous PR

Storing exception frame is extremely prone to circular refernece because it contains the reference to objects.

When tensorizer is not installed, it leaks llm instance because error frame has references to various modules which cause circular reference problem.

I also found spec decoding has a circular reference issue, and I solved it using weakref.proxy.

Co-authored-by: Lei Wen <wenlei03@qiyi.com>

Co-authored-by: SangBin Cho <sangcho@sangcho-LT93GQWG9C.local>