GitLab

On Windows, recent llama.cpp changes make mmap slower in most
cases, so default to off.  This also implements a tri-state for
use_mmap so we can detect the difference between a user provided
value of true/false, or unspecified.

Implement support for GPU env var workarounds, and leverage
this for the Vega RX 56 which needs
HSA_ENABLE_SDMA=0 set to work properly

Still not complete, needs some refinement to our prediction to understand the
discrete GPUs available space so we can see how many layers fit in each one
since we can't split one layer across multiple GPUs we can't treat free space
as one logical block

Critical fix from llama.cpp JSON grammar to forbid un-escaped escape characters inside strings, which breaks parsing. (#3782)

* Revert "use `int32_t` for call to tokenize (#4738)"

This reverts commit 763bb65d.

* Revert "vocab only"

This reverts commit bf54c845.

* Revert "use ffi for tokenizing/detokenizing"

This reverts commit 26a00a04.

* partial offloading: allow flash attention and disable mmap

* allow mmap with num_gpu=0

On some systems, 1 minute isn't sufficient to finish the load after it
hits 100% This creates 2 distinct timers, although they're both set to
the same value for now so we can refine the timeouts further.

Signed-off-by: Lei Jitang <leijitang@outlook.com>

If the client closes the connection before we finish loading the model
we abort, so lets make the log message clearer why to help users
understand this failure mode

This doesn't expose a UX yet, but wires the initial server portion
of progress reporting during load

* put flash attention behind flag for now

* add test

* remove print

* up timeout for sheduler tests

* feat: enable flash attention if supported

* feat: enable flash attention if supported

* feat: enable flash attention if supported

* feat: add flash_attn support

Only dump env vars we care about in the logs

This reverts commit ce3b212d.

* dont clamp ctx size in `PredictServerFit`

* minimum 4 context

* remove context warning

This records more GPU usage information for eventual UX inclusion.

This will bubble up a much more informative error message if noexec
is preventing us from running the subprocess

Trying to live off the land for cuda libraries was not the right strategy.  We need to use the version we compiled against to ensure things work properly

* fix llava models not working after first request

* individual requests only for llava models

This moves all the env var reading into one central module
and logs the loaded config once at startup which should
help in troubleshooting user server logs

log when the waiting for the process to stop to help debug when other tasks execute during this wait.
expire timer clear the timer reference because it will not be reused.
close will clean up expireTimer if calling code has not already done this.