• Graham King's avatar
    feat(dynamo-run): Use llama.cpp as the default engine for GGUF (#1276) · 3e3c3b10
    Graham King authored
    Previously `mistral.rs` was the default engine for both safetensors and GGUF models. Now it is only the default for safetensors, `llama.cpp` becomes the default for GGUF.
    
    Why?
    
    - Since #1177 `llama.cpp` is built-in by default, so we can switch.
    - `llama.cpp` is very very good at running GGUF (but can't run other types of model), so we should switch.
    
    Dynamo's multi-engine support gives us a secret super-power: we can use the best engine for this specific format or model.
    
    We can still run GGUF with mistralrs by doing `out=mistralrs`.
    3e3c3b10
local_model.rs 7.37 KB