"vscode:/vscode.git/clone" did not exist on "2071c3800689cf359ec55c612cefa8b88f409354"
Unverified Commit 61413973 authored by Harry Mellor's avatar Harry Mellor Committed by GitHub
Browse files

Stop testing for slow tokenizers as they will not exist soon (#34235)


Signed-off-by: default avatarHarry Mellor <19981378+hmellor@users.noreply.github.com>
parent 94de8715
......@@ -4,7 +4,6 @@ from typing import _get_protocol_attrs # type: ignore
import pytest
from transformers import (
PreTrainedTokenizer,
PreTrainedTokenizerBase,
PreTrainedTokenizerFast,
)
......@@ -25,10 +24,6 @@ def _assert_tokenizer_like(tokenizer: object):
def test_tokenizer_like_protocol():
tokenizer = get_tokenizer("gpt2", use_fast=False)
assert isinstance(tokenizer, PreTrainedTokenizer)
_assert_tokenizer_like(tokenizer)
tokenizer = get_tokenizer("gpt2", use_fast=True)
assert isinstance(tokenizer, PreTrainedTokenizerFast)
_assert_tokenizer_like(tokenizer)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment