Skip to content

GitLab

Unverified Commit 145c00a4 authored by Matthew Bonanni's avatar Matthew Bonanni Committed by GitHub
Browse files

[Bugfix] change FlashMLA reorder_batch_threshold (#27777) 


Signed-off-by: default avatarMatthew Bonanni <mbonanni@redhat.com>
parent 55011aef
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
vllm/v1/attention/backends/mla/flashmla.py
View file @ 145c00a4
...@@ -71,7 +71,7 @@ class FlashMLAMetadata(MLACommonMetadata[FlashMLADecodeMetadata]): ...@@ -71,7 +71,7 @@ class FlashMLAMetadata(MLACommonMetadata[FlashMLADecodeMetadata]):
class FlashMLAMetadataBuilder(MLACommonMetadataBuilder[FlashMLAMetadata]): class FlashMLAMetadataBuilder(MLACommonMetadataBuilder[FlashMLAMetadata]):
cudagraph_support: ClassVar[AttentionCGSupport] = AttentionCGSupport.UNIFORM_BATCH cudagraph_support: ClassVar[AttentionCGSupport] = AttentionCGSupport.UNIFORM_BATCH
query_len_support: ClassVar[QueryLenSupport] = QueryLenSupport.UNIFORM query_len_support: ClassVar[QueryLenSupport] = QueryLenSupport.UNIFORM
reorder_batch_threshold: int = 512 # process small prefills with decode pathway reorder_batch_threshold: int = 128 # process small prefills with decode pathway
# ^ TODO(matt): tune this # ^ TODO(matt): tune this
def __init__( def __init__(
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment