Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
vllm_cscc
Commits
660470e5
Unverified
Commit
660470e5
authored
Aug 07, 2024
by
xiaobochen123
Committed by
GitHub
Aug 06, 2024
Browse files
[Core] Optimize evictor-v2 performance (#7193)
parent
8d59dbb0
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
4 additions
and
2 deletions
+4
-2
vllm/core/evictor_v2.py
vllm/core/evictor_v2.py
+4
-2
No files found.
vllm/core/evictor_v2.py
View file @
660470e5
...
...
@@ -91,8 +91,9 @@ class LRUEvictor(Evictor):
# at the start of OrderedDict. Loop through all these blocks to
# find the one with maximum number of hashed tokens.
for
_id
,
block
in
self
.
free_table
.
items
():
if
evicted_block
.
last_accessed
>
block
.
last_accessed
or
(
evicted_block
.
last_accessed
==
block
.
last_accessed
and
if
evicted_block
.
last_accessed
<
block
.
last_accessed
:
break
if
(
evicted_block
.
last_accessed
==
block
.
last_accessed
and
evicted_block
.
num_hashed_tokens
<
block
.
num_hashed_tokens
):
evicted_block
=
block
evicted_block_id
=
_id
...
...
@@ -109,6 +110,7 @@ class LRUEvictor(Evictor):
def
update
(
self
,
block_id
:
int
,
last_accessed
:
float
):
self
.
free_table
[
block_id
].
last_accessed
=
last_accessed
self
.
free_table
.
move_to_end
(
block_id
)
def
remove
(
self
,
block_id
:
int
):
if
block_id
not
in
self
.
free_table
:
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment