Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
norm
vllm
Commits
a1c67e6d
Commit
a1c67e6d
authored
Feb 16, 2023
by
Woosuk Kwon
Browse files
Minor
parent
9e68a682
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
3 additions
and
2 deletions
+3
-2
cacheflow/worker/cache_engine.py
cacheflow/worker/cache_engine.py
+3
-2
No files found.
cacheflow/worker/cache_engine.py
View file @
a1c67e6d
...
@@ -20,7 +20,8 @@ class CacheEngine:
...
@@ -20,7 +20,8 @@ class CacheEngine:
dtype
:
torch
.
dtype
,
dtype
:
torch
.
dtype
,
)
->
None
:
)
->
None
:
if
head_size
%
16
!=
0
:
if
head_size
%
16
!=
0
:
raise
ValueError
(
f
'head_size (
{
head_size
}
) must be a multiple of 16.'
)
raise
ValueError
(
f
'head_size (
{
head_size
}
) must be a multiple of 16.'
)
self
.
worker_id
=
worker_id
self
.
worker_id
=
worker_id
self
.
gpu_id
=
gpu_id
self
.
gpu_id
=
gpu_id
...
@@ -40,7 +41,7 @@ class CacheEngine:
...
@@ -40,7 +41,7 @@ class CacheEngine:
self
.
cache_stream
=
torch
.
cuda
.
Stream
(
device
=
gpu_id
)
self
.
cache_stream
=
torch
.
cuda
.
Stream
(
device
=
gpu_id
)
assert
self
.
cache_stream
!=
torch
.
cuda
.
current_stream
(
device
=
gpu_id
)
assert
self
.
cache_stream
!=
torch
.
cuda
.
current_stream
(
device
=
gpu_id
)
# Initialize the events for stream synchronization.
# Initialize the events for stream synchronization.
self
.
events
=
[
torch
.
cuda
.
Event
()
for
_
in
range
(
self
.
num_layers
)]
self
.
events
=
[
torch
.
cuda
.
Event
()
for
_
in
range
(
num_layers
)]
def
get_key_block_shape
(
self
)
->
Tuple
[
int
,
int
,
int
,
int
]:
def
get_key_block_shape
(
self
)
->
Tuple
[
int
,
int
,
int
,
int
]:
element_size
=
torch
.
tensor
([],
dtype
=
self
.
dtype
).
element_size
()
element_size
=
torch
.
tensor
([],
dtype
=
self
.
dtype
).
element_size
()
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment