GitLab

default head_kv to 1

This also moves the checkServerHeartbeat call out of the "RunE" Cobra
stuff (that's the only word I have for that) to on-site where it's after
the check for OLLAMA_MODELS, which allows the helpful error message to
be printed before the server heartbeat check. This also arguably makes
the code more readable without the magic/superfluous "pre" function
caller.

Fix numgpu opt miscomparison

Co-authored-by: Pier Francesco Contino <pfcontino@gmail.com>

fix metal gpu

Bump llama.cpp to b2581

Switch back to subprocessing for llama.cpp

Leaving the cudart library loaded kept ~30m of memory
pinned in the GPU in the main process.  This change ensures
we don't hold GPU resources when idle.

We may have users that run into problems with our current
payload model, so this gives us an escape valve.

"cudart init failure: 35" isn't particularly helpful in the logs.

Cleaner shutdown logic, a bit of response hardening

This should resolve a number of memory leak and stability defects by allowing
us to isolate llama.cpp in a separate process and shutdown when idle, and
gracefully restart if it has problems.  This also serves as a first step to be
able to run multiple copies to support multiple models concurrently.

update memory estimations for gpu offloading

refactor model parsing

fix generate output

count each layer independently when deciding gpu offloading

Corrective Retrieval Augmented Generation Demo, powered by Langgraph and Streamlit 🤗

Support: 
- Ollama
- OpenAI APIs

Plugins list updated

* Community Integration: ChatOllama

* fixed typo

Co-authored-by: Michael Yang <mxyng@pm.me>

CI automation for tagging latest images

Bump ROCm to 6.0.2 patch release

CI windows gpu builds

If we're doing generate, test windows cuda and rocm as well