GitLab

Critical fix from llama.cpp JSON grammar to forbid un-escaped escape characters inside strings, which breaks parsing. (#3782)

* fix embedding by adding fixes from llama.cpp upstream

* remove assert

---------
Co-authored-by: Jesper Ek <deadbeef84@gmail.com>

This follows the same pattern for cuda and rocm to allow
disabling the build even when we detect the dependent libraries

* Revert "use `int32_t` for call to tokenize (#4738)"

This reverts commit 763bb65d.

* Revert "vocab only"

This reverts commit bf54c845.

* Revert "use ffi for tokenizing/detokenizing"

This reverts commit 26a00a04.

* use `int32_t` for call to tokenize

* variable naming

* cleanup

* fix crash

* partial offloading: allow flash attention and disable mmap

* allow mmap with num_gpu=0

* update llama.cpp submodule to `5921b8f089d3b7bda86aac5a66825df6a6c10603`

* add patch

On some systems, 1 minute isn't sufficient to finish the load after it
hits 100% This creates 2 distinct timers, although they're both set to
the same value for now so we can refine the timeouts further.

Signed-off-by: Lei Jitang <leijitang@outlook.com>

If the client closes the connection before we finish loading the model
we abort, so lets make the log message clearer why to help users
understand this failure mode

Co-authored-by: Bruce MacDonald <brucewmacdonald@gmail.com>

This doesn't expose a UX yet, but wires the initial server portion
of progress reporting during load

Co-authored-by: ManniX-ITA <20623405+mann1x@users.noreply.github.com>

* put flash attention behind flag for now

* add test

* remove print

* up timeout for sheduler tests

* feat: enable flash attention if supported

* feat: enable flash attention if supported

* feat: enable flash attention if supported

* feat: add flash_attn support