Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
ktransformers
Commits
f5287e90
Commit
f5287e90
authored
Apr 22, 2025
by
qiyuxinlin
Browse files
fix no balance_serve import error
parent
03a65d6b
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
5 additions
and
2 deletions
+5
-2
ktransformers/models/custom_cache.py
ktransformers/models/custom_cache.py
+5
-2
No files found.
ktransformers/models/custom_cache.py
View file @
f5287e90
...
@@ -12,7 +12,10 @@ import torch.nn as nn
...
@@ -12,7 +12,10 @@ import torch.nn as nn
import
transformers
import
transformers
from
transformers
import
Cache
,
PretrainedConfig
from
transformers
import
Cache
,
PretrainedConfig
from
typing
import
List
,
Optional
,
Dict
,
Any
,
Tuple
from
typing
import
List
,
Optional
,
Dict
,
Any
,
Tuple
from
ktransformers.server.balance_serve.settings
import
sched_ext
try
:
from
ktransformers.server.balance_serve.settings
import
sched_ext
except
:
print
(
"no balance_serve"
)
class
StaticCache
(
transformers
.
StaticCache
):
class
StaticCache
(
transformers
.
StaticCache
):
"""
"""
Static Cache class to be used with `torch.compile(model)`.
Static Cache class to be used with `torch.compile(model)`.
...
@@ -210,7 +213,7 @@ class KDeepSeekV3Cache(nn.Module):
...
@@ -210,7 +213,7 @@ class KDeepSeekV3Cache(nn.Module):
self
.
v_caches
=
[]
self
.
v_caches
=
[]
def
load
(
self
,
inference_context
:
sched_ext
.
InferenceContext
):
def
load
(
self
,
inference_context
:
"
sched_ext.InferenceContext
"
):
for
i
in
range
(
self
.
config
.
num_hidden_layers
):
for
i
in
range
(
self
.
config
.
num_hidden_layers
):
self
.
k_caches
.
append
(
self
.
k_caches
.
append
(
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment