Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
ModelZoo
LLama_fastertransformer
Commits
d8f03ffe
Commit
d8f03ffe
authored
Sep 06, 2023
by
zhuwenwen
Browse files
add llama/llama_tokenizer.py
parent
a0382aa6
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
12 additions
and
0 deletions
+12
-0
examples/cpp/llama/llama_tokenizer.py
examples/cpp/llama/llama_tokenizer.py
+12
-0
No files found.
examples/cpp/llama/llama_tokenizer.py
0 → 100644
View file @
d8f03ffe
import
transformers
import
torch
from
transformers
import
LlamaForCausalLM
,
LlamaTokenizer
tokenizer
=
LlamaTokenizer
.
from_pretrained
(
'/data/models/llama-7b-hf/'
)
with
open
(
'./out'
,
'r'
)
as
file
:
output
=
file
.
read
()
output
=
torch
.
tensor
([
int
(
id_
)
for
id_
in
output
.
split
()])
output
=
tokenizer
.
batch_decode
(
output
.
unsqueeze
(
0
),
skip_special_tokens
=
True
,
clean_up_tokenization_spaces
=
False
)[
0
]
print
(
output
)
\ No newline at end of file
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment