[Bugfix] Fix guidance backend for Qwen models (#16210)

Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>

[Bugfix] Fix guidance backend for Qwen models (#16210)
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
3147586e · Benjamin Chislett · GitHub · ed636d99 · 3147586e
Unverified Commit 3147586e authored Apr 07, 2025 by Benjamin Chislett Committed by GitHub Apr 07, 2025
Hide whitespace changes
Inline Side-by-side

Showing with 2 additions and 1 deletion

vllm/v1/structured_output/backend_guidance.py vllm/v1/structured_output/backend_guidance.py +2 -1

No files found.
--- a/vllm/v1/structured_output/backend_guidance.py
+++ b/vllm/v1/structured_output/backend_guidance.py
@@ -46,7 +46,8 @@ class GuidanceBackend(StructuredOutputBackend):
            in vllm_config.decoding_config.guided_decoding_backend)

        tokenizer = tokenizer_group.get_lora_tokenizer(None)
-        self.ll_tokenizer = llguidance_hf.from_tokenizer(tokenizer, None)
+        self.ll_tokenizer = llguidance_hf.from_tokenizer(
+            tokenizer, self.vocab_size)

    def compile_grammar(self, request_type: StructuredOutputOptions,
                        grammar_spec: str) -> StructuredOutputGrammar: