• Graham King's avatar
    feat: Add a mistralrs engine to tio (#178) · 2f700421
    Graham King authored
    This allows us to run a real model.
    
    Build:
    ```
    cargo build --release --features mistralrs,cuda
    ```
    
    Run:
    ```
    ./target/release/tio in=text out=mistralrs --model-path Llama-3.2-1B-Instruct-Q4_K_M.gguf
    ```
    
    Why [mistral.rs](https://github.com/EricLBuehler/mistral.rs)?
    
    - It has no dependencies. You don't need a container or a virtual env to get started.
    - It supports CUDA, Metal (MacOS) and CPU-only. Everyone can join the AI revolution.
    - It starts fast and serves fast (with CUDA). That makes it fun to experiment with.
    - It runs many models, not just Mistral, that's just it's name.
    2f700421
Cargo.lock 149 KB