Unverified Commit 805de738 authored by Ikko Eltociear Ashimine's avatar Ikko Eltociear Ashimine Committed by GitHub
Browse files

Fix typo in tokenizer.py (#750)

conjuction -> conjunction
parent 1b151ed1
...@@ -72,7 +72,7 @@ def detokenize_incrementally( ...@@ -72,7 +72,7 @@ def detokenize_incrementally(
new_token_id: int, new_token_id: int,
skip_special_tokens: bool, skip_special_tokens: bool,
) -> Tuple[str, str]: ) -> Tuple[str, str]:
"""Detokenizes the new token in conjuction with the previous output tokens. """Detokenizes the new token in conjunction with the previous output tokens.
NOTE: This function does not update prev_output_tokens. NOTE: This function does not update prev_output_tokens.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment