Unverified Commit faa7a5da authored by lianyibo's avatar lianyibo Committed by GitHub
Browse files

[Bugfix] Fix unable to run encoder model when disable_hybrid_kv_cache_manager is true (#24571)


Signed-off-by: default avatarlianyibo <lianyibo1@kunlunit.com>
Co-authored-by: default avatarChen Zhang <zhangch99@outlook.com>
parent 56793995
......@@ -754,6 +754,10 @@ def is_kv_cache_type_uniform(kv_cache_spec: dict[str, KVCacheSpec]) -> bool:
True if all layers have the same type, False otherwise.
"""
if not kv_cache_spec:
# Encoder-only models do not have KV cache, kv_cache_type can be
# regarded as uniform.
return True
try:
kv_cache_spec_values = list(kv_cache_spec.values())
_ = kv_cache_spec_values[0].merge(kv_cache_spec_values)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment