1. 29 May, 2025 1 commit
    • Graham King's avatar
      feat: Initial Granite support (#1271) · 7d0c9386
      Graham King authored
      - Add Granite to our tokenizer
      - Fix pre-processor to load context length correctly
      - Add strftime_now Jinja function for prompt templates
      - Update llama.cpp
      - Handle trtllm errors when not using trtllm
      
      Support depends on the engine:
      
      - `mistral.rs`, our default engine, doesn't support Granite yet.
      
      - `llama.cpp` does and works very well:
      ```
      dynamo-run out=llamacpp ~/llms/granite-3.3-2b-instruct-Q4_K_M.gguf --context-length 16384
      ```
      
      - `vllm` also works very well:
      ```
      dynamo-run in=http out=vllm ~/llms/granite-3.3-2b-instruct --context-length 16384
      ```
      
      - `sglang` mostly works, but it doesn't catch the stop token, so we do in the HTTP ingress, and log an error. The Text ingress doesn't catch it because I disabled it to make the raw echo engine work. A bit of work to do here.
      
      Closes: #1245 
      7d0c9386
  2. 09 May, 2025 1 commit
  3. 08 May, 2025 1 commit
    • Graham King's avatar
      feat: Qwen3, Gemma3 and Llama4 support (#1002) · ceaeba3e
      Graham King authored
      . New mistralrs and llamacpp version
      . mistralrs: Handle Gemma 3 and Llama 4 as vision models
      . Update the dynamo-run docs to use Qwen 3
      . Our pre-processor now supports Llama 4's newer multi-modal `config.json`
      . Upgrade minijinja to handle Qwen 3's prompt template
      
      For Llama 4 we'll need to limit the max seq len. vllm says:
      > To serve at least one request with the models's max seq len (10485760), (240.00 GiB KV cache is needed,...
      
      I was able to run Llama 4 with llamacpp and a quantized GGUF, with Dynamo doing the pre-processing.
      ceaeba3e
  4. 24 Mar, 2025 1 commit