• drbh's avatar
    Handle images in chat api (#1828) · c99ecd77
    drbh authored
    This PR allows for messages to be formatted as simple strings, or as an
    array of objects including image urls. This is done by formatting
    content arrays into a simple string.
    
    Example using `llava-hf/llava-v1.6-mistral-7b-hf` 
    
    ```bash
    curl localhost: 3000/v1/chat/completions \
    -X POST \
    -H 'Content-Type: application/json' \
    -d '{
        "model": "tgi",
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "Whats in this image?"
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rabbit.png"
                        }
                    }
                ]
            }
        ],
        "stream": false,
        "max_tokens": 20,
        "seed": 42
    }'
    ```
    
    is equivlant to this more simple request
    
    ```bash
    curl localhost: 3000/v1/chat/completions \
    -X POST \
    -H 'Content-Type: application/json' \
    -d '{
        "model": "tgi",
        "messages": [
            {
                "role": "user",
                "content": "Whats in this image?\n![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rabbit.png)
    
    "
            }
        ],
        "stream": false,
        "max_tokens": 20,
        "seed": 42
    }'
    ```
    
    output
    ```
    # {"id":"","object":"text_completion","created":1714406985,"model":"llava-hf/llava-v1.6-mistral-7b-hf","system_fingerprint":"2.0.1-native","choices":[{"index":0,"message":{"role":"assistant","content":" This is an illustration of an anthropomorphic rabbit in a spacesuit, standing on what"},"logprobs":null,"finish_reason":"length"}],"usage":{"prompt_tokens":2945,"completion_tokens":20,"total_tokens":2965}}%
    ```
    
    ---------
    Co-authored-by: default avatarNicolas Patry <patry.nicolas@protonmail.com>
    c99ecd77
lib.rs 38.3 KB