[Misc] Refactor tokenizer interface (#29693)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Misc] Refactor tokenizer interface (#29693)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
34a98427 · Cyrus Leung · GitHub · f223ed41 · 34a98427 · 34a98427
Unverified Commit 34a98427 authored Nov 29, 2025 by Cyrus Leung Committed by GitHub Nov 29, 2025
17 changed files
--- a/tests/tokenization/test_tokenizer_registry.py
+++ b/tests/tokenization/test_tokenizer_registry.py
--- a/vllm/tokenizers/registry.py
+++ b/vllm/tokenizers/registry.py
--- a/vllm/transformers_utils/config.py
+++ b/vllm/transformers_utils/config.py
--- a/vllm/transformers_utils/detokenizer_utils.py
+++ b/vllm/transformers_utils/detokenizer_utils.py
--- a/vllm/transformers_utils/gguf_utils.py
+++ b/vllm/transformers_utils/gguf_utils.py
--- a/vllm/transformers_utils/tokenizer.py
+++ b/vllm/transformers_utils/tokenizer.py
--- a/vllm/transformers_utils/tokenizer_base.py
+++ b/vllm/transformers_utils/tokenizer_base.py
--- a/vllm/transformers_utils/tokenizers/__init__.py
+++ b/vllm/transformers_utils/tokenizers/__init__.py
--- a/vllm/v1/engine/async_llm.py
+++ b/vllm/v1/engine/async_llm.py
--- a/vllm/v1/engine/detokenizer.py
+++ b/vllm/v1/engine/detokenizer.py
--- a/vllm/v1/engine/input_processor.py
+++ b/vllm/v1/engine/input_processor.py
--- a/vllm/v1/engine/llm_engine.py
+++ b/vllm/v1/engine/llm_engine.py
--- a/vllm/v1/engine/logprobs.py
+++ b/vllm/v1/engine/logprobs.py
--- a/vllm/v1/engine/output_processor.py
+++ b/vllm/v1/engine/output_processor.py
--- a/vllm/v1/structured_output/backend_types.py
+++ b/vllm/v1/structured_output/backend_types.py
--- a/vllm/v1/structured_output/backend_xgrammar.py
+++ b/vllm/v1/structured_output/backend_xgrammar.py
--- a/vllm/v1/structured_output/utils.py
+++ b/vllm/v1/structured_output/utils.py