Unverified Commit f1a1eb4a authored by Yih-Dar's avatar Yih-Dar Committed by GitHub
Browse files

Fix `llama` tokenization doctest (#24990)



fix
Co-authored-by: default avatarydshieh <ydshieh@users.noreply.github.com>
parent a7d21318
...@@ -332,7 +332,7 @@ class LlamaTokenizer(PreTrainedTokenizer): ...@@ -332,7 +332,7 @@ class LlamaTokenizer(PreTrainedTokenizer):
return output return output
def _build_conversation_input_ids(self, conversation: "Conversation") -> List[int]: def _build_conversation_input_ids(self, conversation: "Conversation") -> List[int]:
"""Builds the input ids for a conversation. r"""Builds the input ids for a conversation.
This is the format used in the provided examples. System prompts should be manually added at the beginning of This is the format used in the provided examples. System prompts should be manually added at the beginning of
the conversation. If no system prompt is given, the `DEFAULT_SYSTEM_PROMPT` will be used. the conversation. If no system prompt is given, the `DEFAULT_SYSTEM_PROMPT` will be used.
``` ```
...@@ -347,7 +347,7 @@ class LlamaTokenizer(PreTrainedTokenizer): ...@@ -347,7 +347,7 @@ class LlamaTokenizer(PreTrainedTokenizer):
>>> Conversation( >>> Conversation(
... "<<SYS>>\n Only answer with emojis, and charades\n<</SYS>>\n\nHow can I build a house in 10 septs?" ... "<<SYS>>\n Only answer with emojis, and charades\n<</SYS>>\n\nHow can I build a house in 10 septs?"
... ) ... ) # doctest: +IGNORE_RESULT
``` ```
Args: Args:
conversation (`Conversation`): conversation (`Conversation`):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment