Commit 2ab78325 authored by Aymeric Augustin's avatar Aymeric Augustin
Browse files

Fix F821 flake8 warning (x47).

Ignore warnings related to Python 2, because it's going away soon.
parent 631be270
......@@ -156,7 +156,7 @@ class XLNetTokenizer(PreTrainedTokenizer):
"""
text = self.preprocess_text(text)
# note(zhiliny): in some systems, sentencepiece only accepts str for py2
if six.PY2 and isinstance(text, unicode):
if six.PY2 and isinstance(text, unicode): # noqa: F821
text = text.encode("utf-8")
if not sample:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment