• Graham King's avatar
    feat(tio): Distributed inference! (#235) · 32a748e4
    Graham King authored
    Add support in tio for distributed components and discovery.
    
    Node 1:
    ```
    tio in=http out=tdr://ns/backend/mistralrs
    ```
    
    Node 2:
    ```
    tio in=tdr://ns/backend/mistralrs out=mistralrs ~/llm_models/Llama-3.2-3B-Instruct
    ```
    
    This will use etcd to auto-discover the model and NATS to talk to it. You can run multiple workers on the same endpoint and it will pick one at random each time.
    
    The `ns/backend/mistralrs` are purely symbolic, pick anything as long as it has three parts, and it matches the other node.
    32a748e4
This project manages its dependencies using Cargo. Learn more
Cargo.toml 2.16 KB