Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
18a879f4
Commit
18a879f4
authored
Dec 16, 2019
by
Lysandre
Browse files
fix #2180
parent
d8034092
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
5 additions
and
1 deletion
+5
-1
examples/run_generation.py
examples/run_generation.py
+5
-1
No files found.
examples/run_generation.py
View file @
18a879f4
...
...
@@ -247,7 +247,11 @@ def main():
out
=
out
[:,
len
(
context_tokens
):].
tolist
()
for
o
in
out
:
text
=
tokenizer
.
decode
(
o
,
clean_up_tokenization_spaces
=
True
)
text
=
text
[:
text
.
find
(
args
.
stop_token
)
if
args
.
stop_token
else
None
]
if
args
.
stop_token
:
index
=
text
.
find
(
args
.
stop_token
)
if
index
==
-
1
:
index
=
None
text
=
text
[:
index
]
print
(
text
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment