Commit 96edd990 authored by dcuai's avatar dcuai
Browse files

Update constants.py

parent e187d8b6
import json
with open("/public/home/zhaoying1/work/TencentPretrain-main/models/llama_special_tokens_map.json", mode="r", encoding="utf-8") as f:
with open("./models/llama_special_tokens_map.json", mode="r", encoding="utf-8") as f:
special_tokens_map = json.load(f)
UNK_TOKEN = special_tokens_map["unk_token"]
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment