Skip to content

GitLab

Commit 72501097 authored by xuxzh1's avatar xuxzh1 🎱
Browse files

update rocm.py

parent d4bccff3
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 3 deletions
server/text_generation_server/layers/attention/rocm.py
View file @ 72501097
...@@ -80,7 +80,6 @@ def paged_attention( ...@@ -80,7 +80,6 @@ def paged_attention(
_PARTITION_SIZE = _PARTITION_SIZE_V1V2 _PARTITION_SIZE = _PARTITION_SIZE_V1V2
else: else:
_PARTITION_SIZE = _PARTITION_SIZE_CUSTOM _PARTITION_SIZE = _PARTITION_SIZE_CUSTOM
_PARTITION_SIZE = 512
max_num_partitions = (max_s + _PARTITION_SIZE - 1) // _PARTITION_SIZE max_num_partitions = (max_s + _PARTITION_SIZE - 1) // _PARTITION_SIZE
input_lengths = seqlen.input_lengths + seqlen.cache_lengths input_lengths = seqlen.input_lengths + seqlen.cache_lengths
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment