Commit df978fdd authored by cclauss's avatar cclauss Committed by Taylor Robie
Browse files

Use six and feature detection in string conversion (#4740)

* Use six and feature detection in string conversion

Leverage [__six.ensure_text()__](https://github.com/benjaminp/six/blob/master/six.py#L890) to deliver Unicode text in both Python 2 and Python 3.

Follow Python porting best practice [use feature detection instead of version detection](https://docs.python.org/3/howto/pyporting.html#use-feature-detection-instead-of-version-detection) in ___unicode_to_native()__.

* Revert the use of six.ensure_text()

Thanks for catching that!  I jumped the gun.  It is I who have brought shame...
parent 75150327
......@@ -202,17 +202,17 @@ def _load_vocab_file(vocab_file, reserved_tokens=None):
def _native_to_unicode(s):
"""Convert string to unicode (required in Python 2)."""
if six.PY2:
return s if isinstance(s, unicode) else s.decode("utf-8") # pylint: disable=undefined-variable
else:
try: # Python 2
return s if isinstance(s, unicode) else s.decode("utf-8")
except NameError: # Python 3
return s
def _unicode_to_native(s):
"""Convert string from unicode to native format (required in Python 2)."""
if six.PY2:
return s.encode("utf-8") if isinstance(s, unicode) else s # pylint: disable=undefined-variable
else:
try: # Python 2
return s.encode("utf-8") if isinstance(s, unicode) else s
except NameError: # Python 3
return s
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment