"vscode:/vscode.git/clone" did not exist on "344c21dc0a0fd99a6653be51268bee5ff5db0d0d"
  • Graham King's avatar
    feat(dynamo-run): Allow setting KV cache block size (#1175) · 183f2b32
    Graham King authored
    Example:
    ```
    dynamo-run out=<engine> <model> --kv-cache-block-size 64
    ```
    
    In a distributed system this goes on the worker node and is propagated to ingress via the model deployment card.
    
    Previously hard coded to 16, which is now the default.
    
    - Load context_length from model. Closes #1172
    - Store context length and KV cache block size in Model Deployment Card #1170
    183f2b32
lib.rs 11.5 KB