Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
change
sglang
Commits
f39a0197
"vscode:/vscode.git/clone" did not exist on "a1b5b42af428b9a6352074dc3fbb1c520c324892"
Unverified
Commit
f39a0197
authored
Sep 24, 2024
by
Ying Sheng
Committed by
GitHub
Sep 24, 2024
Browse files
Revert "kernel: use tensor cores for flashinfer gqa kernels" (#1511)
parent
3c93187c
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
3 additions
and
11 deletions
+3
-11
python/sglang/srt/layers/attention_backend.py
python/sglang/srt/layers/attention_backend.py
+3
-11
No files found.
python/sglang/srt/layers/attention_backend.py
View file @
f39a0197
...
@@ -86,17 +86,9 @@ class FlashInferAttnBackend(AttentionBackend):
...
@@ -86,17 +86,9 @@ class FlashInferAttnBackend(AttentionBackend):
super
().
__init__
()
super
().
__init__
()
self
.
model_runner
=
model_runner
self
.
model_runner
=
model_runner
local_num_qo_heads
=
(
if
not
_grouped_size_compiled_for_decode_kernels
(
model_runner
.
model_config
.
num_attention_heads
//
model_runner
.
tp_size
model_runner
.
model_config
.
num_attention_heads
//
model_runner
.
tp_size
,
)
model_runner
.
model_config
.
get_num_kv_heads
(
model_runner
.
tp_size
),
local_num_kv_heads
=
model_runner
.
model_config
.
get_num_kv_heads
(
model_runner
.
tp_size
)
if
(
not
_grouped_size_compiled_for_decode_kernels
(
local_num_qo_heads
,
local_num_kv_heads
)
or
local_num_qo_heads
//
local_num_kv_heads
>
4
):
):
self
.
decode_use_tensor_cores
=
True
self
.
decode_use_tensor_cores
=
True
else
:
else
:
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment