"git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "d73baeebc52a12316cea0821b589c0b40e3768a9"
Unverified Commit dced2624 authored by Dat Quoc Nguyen's avatar Dat Quoc Nguyen Committed by GitHub
Browse files

Update tokenization_bertweet.py (#16941)

The emoji version must be either 0.5.4 or 0.6.0. Newer emoji versions have been updated to newer versions of the Emoji Charts, thus not consistent with the one used for pre-processing the pre-training Tweet corpus (i.e. not consistent with the vocab).
parent 992996e9
......@@ -152,7 +152,7 @@ class BertweetTokenizer(PreTrainedTokenizer):
self.demojizer = demojize
except ImportError:
logger.warning(
"emoji is not installed, thus not converting emoticons or emojis into text. Please install emoji: pip3 install emoji"
"emoji is not installed, thus not converting emoticons or emojis into text. Install emoji: pip3 install emoji==0.6.0"
)
self.demojizer = None
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment