Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
__init__.py 335 Bytes
Newer Older
Cyrus Leung's avatar
[Misc] Refactor tokenizer interface (#29693)  
Cyrus Leung committed
1
2
3 
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
Cyrus Leung's avatar
[Misc] Update `TokenizerLike` interface and move `get_cached_tokenizer` (#29730)  
Cyrus Leung committed
4 
from .hf import HfTokenizer
Cyrus Leung's avatar
[Misc] Refactor tokenizer interface (#29693)  
Cyrus Leung committed
5
6
7
8 
from .mistral import MistralTokenizer
from .protocol import TokenizerLike
from .registry import TokenizerRegistry
Cyrus Leung's avatar
[Misc] Update `TokenizerLike` interface and move `get_cached_tokenizer` (#29730)  
Cyrus Leung committed
9 
__all__ = ["TokenizerLike", "HfTokenizer", "MistralTokenizer", "TokenizerRegistry"]