Skip to content

GitLab

Commit 2ab78325 authored by Aymeric Augustin's avatar Aymeric Augustin
Browse files

Fix F821 flake8 warning (x47). 

Ignore warnings related to Python 2, because it's going away soon.
parent 631be270
Show whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
transformers/tokenization_xlnet.py
View file @ 2ab78325
...@@ -156,7 +156,7 @@ class XLNetTokenizer(PreTrainedTokenizer): ...@@ -156,7 +156,7 @@ class XLNetTokenizer(PreTrainedTokenizer):
""" """
text = self.preprocess_text(text) text = self.preprocess_text(text)
# note(zhiliny): in some systems, sentencepiece only accepts str for py2 # note(zhiliny): in some systems, sentencepiece only accepts str for py2
if six.PY2 and isinstance(text, unicode): if six.PY2 and isinstance(text, unicode): # noqa: F821
text = text.encode("utf-8") text = text.encode("utf-8")
if not sample: if not sample:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment