Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
ComfyUI
Commits
88ed8930
Commit
88ed8930
authored
Jul 23, 2024
by
comfyanonymous
Browse files
Allow SPieceTokenizer to load model from a byte string.
parent
334ba48c
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
6 additions
and
3 deletions
+6
-3
comfy/text_encoders/spiece_tokenizer.py
comfy/text_encoders/spiece_tokenizer.py
+6
-3
No files found.
comfy/text_encoders/spiece_tokenizer.py
View file @
88ed8930
import
os
class
SPieceTokenizer
:
add_eos
=
True
@
staticmethod
def
from_pretrained
(
path
):
return
SPieceTokenizer
(
path
)
def
__init__
(
self
,
tokenizer_path
):
import
sentencepiece
self
.
tokenizer
=
sentencepiece
.
SentencePieceProcessor
(
model_file
=
tokenizer_path
)
self
.
end
=
self
.
tokenizer
.
eos_id
()
if
isinstance
(
tokenizer_path
,
bytes
):
self
.
tokenizer
=
sentencepiece
.
SentencePieceProcessor
(
model_proto
=
tokenizer_path
,
add_eos
=
self
.
add_eos
)
else
:
self
.
tokenizer
=
sentencepiece
.
SentencePieceProcessor
(
model_file
=
tokenizer_path
,
add_eos
=
self
.
add_eos
)
def
get_vocab
(
self
):
out
=
{}
...
...
@@ -18,5 +22,4 @@ class SPieceTokenizer:
def
__call__
(
self
,
string
):
out
=
self
.
tokenizer
.
encode
(
string
)
out
+=
[
self
.
end
]
return
{
"input_ids"
:
out
}
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment