• drbh's avatar
    Outlines guided generation (#1539) · cef0553d
    drbh authored
    This WIP PR starts to add grammar support via outlines, currently this
    PR supports very simple regex grammars and does not optimize for
    precompiling or caching grammar fsm's.
    
    todo:
    - [X] add simple outlines guidance to `NextTokenChooser`
    - [X] update protos for grammar
    - [X] update generation params API
    - [X] constrain simple grammar
    - [ ] support parsing more complex grammar into fsm
    - [ ] support all outline support grammar types
    - [ ] explore optimizations to avoid recompiling grammars
    
    guided request
    ```bash
    curl -s 'http://localhost:3000/generate' \
    --header 'Content-Type: application/json' \
    --data-raw '{
        "inputs": "make an email for david: \n",
        "parameters": {
            "max_new_tokens": 6,
            "grammar": "[\\w-]+@([\\w-]+\\.)+[\\w-]+"
        }
    }' | jq
    ```
    response
    ```json
    {
      "generated_text": "david@example.com"
    }
    ```
    
    unguided request
    ```bash
    curl -s 'http://localhost:3000/generate' \
    --header 'Content-Type: application/json' \
    --data '{
        "inputs": "make an email for david: \n",
        "parameters": {
            "max_new_tokens": 6
        }
    }' | jq
    ```
    response
    ```json
    {
      "generated_text": "    email = 'david"
    }
    ```
    cef0553d
client.rs 8.01 KB