GitLab

Co-authored-by: caishangming.csm <caishangming.csm@alibaba-inc.com>

[Speculative Decoding] Enabling bonus token in speculative decoding for KV cache based models (#5765)

Storing exception frame is extremely prone to circular refernece because it contains the reference to objects.

When tensorizer is not installed, it leaks llm instance because error frame has references to various modules which cause circular reference problem.

I also found spec decoding has a circular reference issue, and I solved it using weakref.proxy.

Co-authored-by: Lei Wen <wenlei03@qiyi.com>