"git@developer.sourcefind.cn:hehl2/torchaudio.git" did not exist on "7ab08200a810153a55b6c3090a8b525b83040b54"
Commit 28e64ad5 authored by Morgan Funtowicz's avatar Morgan Funtowicz
Browse files

Raise an exception if the pipeline allocator can't determine the tokenizer from the model.

parent be5bf7b8
...@@ -370,11 +370,12 @@ def pipeline(task: str, model, config: Optional[PretrainedConfig] = None, tokeni ...@@ -370,11 +370,12 @@ def pipeline(task: str, model, config: Optional[PretrainedConfig] = None, tokeni
Utility factory method to build pipeline. Utility factory method to build pipeline.
""" """
# Try to infer tokenizer from model name (if provided as str) # Try to infer tokenizer from model name (if provided as str)
if tokenizer is None and isinstance(model, str): if not isinstance(tokenizer, PreTrainedTokenizer):
tokenizer = model if not isinstance(model, str):
else:
# Impossible to guest what is the right tokenizer here # Impossible to guest what is the right tokenizer here
raise Exception('Tokenizer cannot be None if provided model is a PreTrainedModel instance') raise Exception('Tokenizer cannot be None if provided model is a PreTrainedModel instance')
else:
tokenizer = model
tokenizer = tokenizer if isinstance(tokenizer, PreTrainedTokenizer) else AutoTokenizer.from_pretrained(tokenizer) tokenizer = tokenizer if isinstance(tokenizer, PreTrainedTokenizer) else AutoTokenizer.from_pretrained(tokenizer)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment