Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
Lmdeploy
Commits
9d2a48ae
Commit
9d2a48ae
authored
Mar 06, 2024
by
zhouxiang
Browse files
解决当request_output_len+input_id长度大于session_len时就不输出的问题,将策略改成自适应request_output_len,可以生成到session_len为止
parent
2e528580
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
7 additions
and
1 deletion
+7
-1
lmdeploy/serve/async_engine.py
lmdeploy/serve/async_engine.py
+7
-1
No files found.
lmdeploy/serve/async_engine.py
View file @
9d2a48ae
...
@@ -264,12 +264,18 @@ class AsyncEngine:
...
@@ -264,12 +264,18 @@ class AsyncEngine:
prompt
=
self
.
model
.
messages2prompt
(
prompt
,
sequence_start
)
prompt
=
self
.
model
.
messages2prompt
(
prompt
,
sequence_start
)
input_ids
=
self
.
tokenizer
.
encode
(
prompt
,
add_bos
=
sequence_start
)
input_ids
=
self
.
tokenizer
.
encode
(
prompt
,
add_bos
=
sequence_start
)
finish_reason
=
None
finish_reason
=
None
request_output_len
=
min
(
request_output_len
,
self
.
tm_model
.
session_len
-
self
.
id2step
[
str
(
session_id
)]
-
len
(
input_ids
))
request_output_len
=
max
(
0
,
request_output_len
)
if
stop
is
True
:
if
stop
is
True
:
self
.
stop_session
(
session_id
)
self
.
stop_session
(
session_id
)
yield
GenOut
(
''
,
self
.
id2step
[
str
(
session_id
)],
len
(
input_ids
),
0
,
yield
GenOut
(
''
,
self
.
id2step
[
str
(
session_id
)],
len
(
input_ids
),
0
,
finish_reason
)
finish_reason
)
elif
self
.
id2step
[
str
(
session_id
)]
+
len
(
elif
self
.
id2step
[
str
(
session_id
)]
+
len
(
input_ids
)
+
request_output_len
>
=
self
.
tm_model
.
session_len
:
input_ids
)
+
request_output_len
>
self
.
tm_model
.
session_len
:
finish_reason
=
'length'
finish_reason
=
'length'
yield
GenOut
(
''
,
self
.
id2step
[
str
(
session_id
)],
len
(
input_ids
),
0
,
yield
GenOut
(
''
,
self
.
id2step
[
str
(
session_id
)],
len
(
input_ids
),
0
,
finish_reason
)
finish_reason
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment