GitLab

This reverts commit f5f245cc, reversing
changes made to 94d37fdc.

this change broke gguf v2 which is incorrectly detected as big endian

Critical fix from llama.cpp JSON grammar to forbid un-escaped escape characters inside strings, which breaks parsing. (#3782)

* fix embedding by adding fixes from llama.cpp upstream

* remove assert

---------
Co-authored-by: Jesper Ek <deadbeef84@gmail.com>

This follows the same pattern for cuda and rocm to allow
disabling the build even when we detect the dependent libraries

* Revert "use `int32_t` for call to tokenize (#4738)"

This reverts commit 763bb65d.

* Revert "vocab only"

This reverts commit bf54c845.

* Revert "use ffi for tokenizing/detokenizing"

This reverts commit 26a00a04.

* use `int32_t` for call to tokenize

* variable naming

* cleanup

* fix crash

* partial offloading: allow flash attention and disable mmap

* allow mmap with num_gpu=0

* update llama.cpp submodule to `5921b8f089d3b7bda86aac5a66825df6a6c10603`

* add patch

On some systems, 1 minute isn't sufficient to finish the load after it
hits 100% This creates 2 distinct timers, although they're both set to
the same value for now so we can refine the timeouts further.

Signed-off-by: Lei Jitang <leijitang@outlook.com>

If the client closes the connection before we finish loading the model
we abort, so lets make the log message clearer why to help users
understand this failure mode

Co-authored-by: Bruce MacDonald <brucewmacdonald@gmail.com>

This doesn't expose a UX yet, but wires the initial server portion
of progress reporting during load

Co-authored-by: ManniX-ITA <20623405+mann1x@users.noreply.github.com>

* put flash attention behind flag for now

* add test

* remove print

* up timeout for sheduler tests

* feat: enable flash attention if supported

* feat: enable flash attention if supported

* feat: enable flash attention if supported

* feat: add flash_attn support