• Graham King's avatar
    feat(tio): Distributed inference! (#235) · 32a748e4
    Graham King authored
    Add support in tio for distributed components and discovery.
    
    Node 1:
    ```
    tio in=http out=tdr://ns/backend/mistralrs
    ```
    
    Node 2:
    ```
    tio in=tdr://ns/backend/mistralrs out=mistralrs ~/llm_models/Llama-3.2-3B-Instruct
    ```
    
    This will use etcd to auto-discover the model and NATS to talk to it. You can run multiple workers on the same endpoint and it will pick one at random each time.
    
    The `ns/backend/mistralrs` are purely symbolic, pick anything as long as it has three parts, and it matches the other node.
    32a748e4
lib.rs 980 Bytes