"include/gtest/vscode:/vscode.git/clone" did not exist on "75e2713e451b8d51fc185957383f0ea97a40ef15"
  • Graham King's avatar
    feat(tio): Distributed inference! (#235) · 32a748e4
    Graham King authored
    Add support in tio for distributed components and discovery.
    
    Node 1:
    ```
    tio in=http out=tdr://ns/backend/mistralrs
    ```
    
    Node 2:
    ```
    tio in=tdr://ns/backend/mistralrs out=mistralrs ~/llm_models/Llama-3.2-3B-Instruct
    ```
    
    This will use etcd to auto-discover the model and NATS to talk to it. You can run multiple workers on the same endpoint and it will pick one at random each time.
    
    The `ns/backend/mistralrs` are purely symbolic, pick anything as long as it has three parts, and it matches the other node.
    32a748e4
text.rs 5.54 KB