GitLab

Signed-off-by: Lei Jitang <leijitang@outlook.com>

small fix on examples/python-simplechat/client.py to actually get a streamed response and get tokens printed as we receive it (#4671)

Ensure `nvidia` and `nvidia_uvm` kernel modules are loaded in `install.sh` script and at startup (#4652)

* ensure kernel modules are loaded in `install.sh` script and at startup

* indentation

* use `SUDO` variable

* restart if nouveau is detected

* consistent success message for AMD

Report better warning on client closed abort of load

If the client closes the connection before we finish loading the model
we abort, so lets make the log message clearer why to help users
understand this failure mode

Fix download retry issue

fix q5_0, q5_1

Co-authored-by: Bruce MacDonald <brucewmacdonald@gmail.com>

Tidy up developer guide a little

Wire up load progress

This doesn't expose a UX yet, but wires the initial server portion
of progress reporting during load

Co-authored-by: ManniX-ITA <20623405+mann1x@users.noreply.github.com>

* put flash attention behind flag for now

* add test

* remove print

* up timeout for sheduler tests

PreTokenziers -> PreTokenizers

add Ctrl + W shortcut

simplify safetensors reading

Convert directly from llama3

The spelling of the term "request" has been corrected, which was previously mistakenly written as "requeset" in the error log message.

fix quantize file types