Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
AutoAWQ
Commits
7a3d06d6
Commit
7a3d06d6
authored
Oct 06, 2023
by
Casper Hansen
Browse files
Update comment
parent
204a3a12
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
3 additions
and
1 deletion
+3
-1
awq/modules/fused/cache.py
awq/modules/fused/cache.py
+3
-1
No files found.
awq/modules/fused/cache.py
View file @
7a3d06d6
...
...
@@ -26,8 +26,10 @@ class WindowedCache:
def
roll_kv
(
self
,
roll_len
,
start_pos
):
"""
For example, with
roll_len=3 and [A,B,C,D,E] we get [D,E,F,G,H]
With sink=0,
roll_len=3
,
and [A,B,C,D,E] we get [D,E,F,G,H]
With sink=1, roll_len=3, and [A,B,C,D,E] we get [A,E,F,G,H]
With sink=2, roll_len=3, and [A,B,C,D,E] we get [A,B,F,G,H]
With sink=3, roll_len=3, and [A,B,C,D,E] we get [A,B,C,G,H]
"""
# Roll only the necessary part of the cache to the left
self
.
v
[:,
:,
self
.
attention_sinks
:
-
roll_len
+
self
.
attention_sinks
,
:]
=
self
.
v
[:,
:,
roll_len
:,
:]
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment