Skip to content

GitLab

Unverified Commit 3147586e authored by Benjamin Chislett's avatar Benjamin Chislett Committed by GitHub
Browse files

[Bugfix] Fix guidance backend for Qwen models (#16210) 


Signed-off-by: default avatarBenjamin Chislett <benjamin.chislett@centml.ai>
parent ed636d99
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 1 deletion
vllm/v1/structured_output/backend_guidance.py
View file @ 3147586e
...@@ -46,7 +46,8 @@ class GuidanceBackend(StructuredOutputBackend): ...@@ -46,7 +46,8 @@ class GuidanceBackend(StructuredOutputBackend):
in vllm_config.decoding_config.guided_decoding_backend) in vllm_config.decoding_config.guided_decoding_backend)
tokenizer = tokenizer_group.get_lora_tokenizer(None) tokenizer = tokenizer_group.get_lora_tokenizer(None)
self.ll_tokenizer = llguidance_hf.from_tokenizer(tokenizer, None) self.ll_tokenizer = llguidance_hf.from_tokenizer(
tokenizer, self.vocab_size)
def compile_grammar(self, request_type: StructuredOutputOptions, def compile_grammar(self, request_type: StructuredOutputOptions,
grammar_spec: str) -> StructuredOutputGrammar: grammar_spec: str) -> StructuredOutputGrammar:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment