Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
ModelZoo
LLaMA_TencentPretrain_pytorch
Commits
96edd990
Commit
96edd990
authored
Aug 15, 2024
by
dcuai
Browse files
Update constants.py
parent
e187d8b6
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
1 addition
and
1 deletion
+1
-1
tencentpretrain/utils/constants.py
tencentpretrain/utils/constants.py
+1
-1
No files found.
tencentpretrain/utils/constants.py
View file @
96edd990
import
json
import
json
with
open
(
"
/public/home/zhaoying1/work/TencentPretrain-main
/models/llama_special_tokens_map.json"
,
mode
=
"r"
,
encoding
=
"utf-8"
)
as
f
:
with
open
(
"
.
/models/llama_special_tokens_map.json"
,
mode
=
"r"
,
encoding
=
"utf-8"
)
as
f
:
special_tokens_map
=
json
.
load
(
f
)
special_tokens_map
=
json
.
load
(
f
)
UNK_TOKEN
=
special_tokens_map
[
"unk_token"
]
UNK_TOKEN
=
special_tokens_map
[
"unk_token"
]
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment