• drbh's avatar
    feat: conditionally toggle chat on invocations route (#1454) · 98e5faff
    drbh authored
    This PR adds support for reading the `OAI_ENABLED` env var which will
    changes the function called when the `/invocations` is called.
    
    If `OAI_ENABLED=true` the `chat_completions` method is used otherwise it
    defaults to `compat_generate`.
    
    example running the router
    ```bash
    OAI_ENABLED=true \
      cargo run -- \
      --tokenizer-name mistralai/Mistral-7B-Instruct-v0.2
    ```
    
    example request
    ```bash
    curl localhost:3000/invocations \
        -X POST \
        -d '{ "model": "tgi", "messages": [ { "role": "user", "content": "What is the IP address of the Google DNS servers?" } ], "stream": false, "max_tokens": 20, "logprobs": true, "seed": 0 }' \
        -H 'Content-Type: application/json' | jq 
    ```
    
    **please let me know if any naming changes are needed or if any other
    routes need similar functionality.
    98e5faff
main.rs 16.9 KB