Unverified Commit 066ea374 authored by Álvaro Somoza's avatar Álvaro Somoza Committed by GitHub
Browse files

[Tests] Fix ChatGLMTokenizer (#9536)

fix
parent 9cd37557
...@@ -277,6 +277,7 @@ class ChatGLMTokenizer(PreTrainedTokenizer): ...@@ -277,6 +277,7 @@ class ChatGLMTokenizer(PreTrainedTokenizer):
padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
pad_to_multiple_of: Optional[int] = None, pad_to_multiple_of: Optional[int] = None,
return_attention_mask: Optional[bool] = None, return_attention_mask: Optional[bool] = None,
padding_side: Optional[bool] = None,
) -> dict: ) -> dict:
""" """
Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Pad encoded inputs (on left/right and up to predefined length or max length in the batch)
...@@ -298,6 +299,9 @@ class ChatGLMTokenizer(PreTrainedTokenizer): ...@@ -298,6 +299,9 @@ class ChatGLMTokenizer(PreTrainedTokenizer):
pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value.
This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability
`>= 7.5` (Volta). `>= 7.5` (Volta).
padding_side (`str`, *optional*):
The side on which the model should have padding applied. Should be selected between ['right', 'left'].
Default value is picked from the class attribute of the same name.
return_attention_mask: return_attention_mask:
(optional) Set to False to avoid returning attention mask (default: set to model specifics) (optional) Set to False to avoid returning attention mask (default: set to model specifics)
""" """
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment