Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
vllm_cscc
Commits
49eedea3
Unverified
Commit
49eedea3
authored
Mar 18, 2024
by
Antoni Baum
Committed by
GitHub
Mar 18, 2024
Browse files
[Core] Zero-copy asdict for InputMetadata (#3475)
parent
9fdf3de3
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
12 additions
and
4 deletions
+12
-4
vllm/model_executor/input_metadata.py
vllm/model_executor/input_metadata.py
+11
-2
vllm/worker/model_runner.py
vllm/worker/model_runner.py
+1
-2
No files found.
vllm/model_executor/input_metadata.py
View file @
49eedea3
from
dataclasses
import
dataclass
from
typing
import
Optional
from
dataclasses
import
dataclass
,
fields
from
typing
import
Optional
,
Any
,
Dict
import
torch
...
...
@@ -31,3 +31,12 @@ class InputMetadata:
def
__post_init__
(
self
):
# will not appear in the __repr__ and __init__
self
.
attn_bias
=
None
def
asdict_zerocopy
(
self
)
->
Dict
[
str
,
Any
]:
"""Similar to dataclasses.asdict, but avoids deepcopying."""
# Note that if we add dataclasses as fields, they will need
# similar handling.
return
{
field
.
name
:
getattr
(
self
,
field
.
name
)
for
field
in
fields
(
self
)
}
vllm/worker/model_runner.py
View file @
49eedea3
import
contextlib
import
dataclasses
import
time
from
typing
import
Dict
,
List
,
Optional
,
Tuple
,
Set
,
Union
...
...
@@ -527,7 +526,7 @@ class ModelRunner:
"lora_requests"
:
lora_requests
,
"lora_mapping"
:
lora_mapping
,
}
metadata_dict
.
update
(
dataclasses
.
asdict
(
input_metadata
))
metadata_dict
.
update
(
input_metadata
.
asdict_zerocopy
(
))
broadcast_tensor_dict
(
metadata_dict
,
src
=
0
)
else
:
metadata_dict
=
broadcast_tensor_dict
(
src
=
0
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment