GitLab

https://en.wikipedia.org/wiki/Server-sent_events

Adds some bits for first-time users (like me 😄 )

@njhill, @yk FYI

generated_text was concatenated to the user prompt for legacy reason. We
want to remove this behaviour as we don't think it is useful and even
detrimonial to usability.

We also remove the unused Vec.

Co-authored-by: Nick Hill <nickhill@us.ibm.com>

Reverts huggingface/text-generation-inference#36

Add token streaming using ServerSideEvents (SSE).

The signature of the SSE events is: 

```rust
struct Details {
    finish_reason: String,
    generated_tokens: u32,
    seed: Option<u64>,
}

struct StreamResponse {
    token: Token,
    generated_text: Option<String>,
    details: Option<Details>,
}

struct ErrorResponse {
    error: String,
}
```

Co-authored-by: Yannic Kilcher <yk@users.noreply.github.com>

@njhill

- Fix some type hints, in particular base tokenizer class
- Make use of `tensor.new_zero/empty` methods
- Simplify env var string parsing in launcher

Co-authored-by: Nick Hill <nickhill@us.ibm.com>

Fixes #12 in the easiest way I could think of.

I noticed some opportunity to collapse some of the logic, in case you
are interested.

There's currently a discrepancy in the tokenization between the router
and python server code. The latter includes special tokens but former
does not.

This results in a token count mismatch for seq2seq models such as mt0
where the tokenizer emits an EOS token at the end.

This in turn results in some unexpected/incorrect output, in particular
when batch concatenation is involved, because the python code uses the
input length passed from the router for each row.

As far as I can tell, it is better to include this token in the encoder
`input_ids`, so I guess it's best to just adjust on the router side.

AFAIK there is no torch device type called "gpu".