GitLab

Bump llama.cpp to b2581

Switch back to subprocessing for llama.cpp

Leaving the cudart library loaded kept ~30m of memory
pinned in the GPU in the main process.  This change ensures
we don't hold GPU resources when idle.

We may have users that run into problems with our current
payload model, so this gives us an escape valve.

"cudart init failure: 35" isn't particularly helpful in the logs.

Cleaner shutdown logic, a bit of response hardening

This should resolve a number of memory leak and stability defects by allowing
us to isolate llama.cpp in a separate process and shutdown when idle, and
gracefully restart if it has problems.  This also serves as a first step to be
able to run multiple copies to support multiple models concurrently.

update memory estimations for gpu offloading

refactor model parsing

fix generate output

count each layer independently when deciding gpu offloading

Corrective Retrieval Augmented Generation Demo, powered by Langgraph and Streamlit 🤗

Support: 
- Ollama
- OpenAI APIs

Plugins list updated

* Community Integration: ChatOllama

* fixed typo

Co-authored-by: Michael Yang <mxyng@pm.me>

CI automation for tagging latest images

Bump ROCm to 6.0.2 patch release

CI windows gpu builds

If we're doing generate, test windows cuda and rocm as well

fix: trim quotes on OLLAMA_ORIGINS

fix: workflows

only generate on changes to llm subdirectory