GitLab

Harden for zero detected GPUs

At least with the ROCm libraries, its possible to have the library
present with zero GPUs.  This fix avoids a divide by zero bug in llm.go
when we try to calculate GPU memory with zero GPUs.

Add support for CUDA 5.0 cards

ROCm: Correct the response string in rocm_get_version function

Detect lack of AVX and fallback to CPU mode

We build the GPU libraries with AVX enabled to ensure that if not all
layers fit on the GPU we get better performance in a mixed mode.
If the user is using a virtualization/emulation system that lacks AVX
this used to result in an illegal instruction error and crash before this
fix.  Now we will report a warning in the server log, and just use
CPU mode to ensure we don't crash.

fix build

download: add inactivity monitor

Add back ROCm container support

Ignore AMD integrated GPUs

Fix crash on cuda ml init failure

This adds ROCm support back as a discrete image.

Detect and ignore integrated GPUs reported by rocm.

The new driver lookup code was triggering after init failure due to a missing return

Switch back to ubuntu base

The size increase for rocm support in the standard image is problematic
We'll revisit multiple tags for rocm support in a follow up PR.

build cuda and rocm

* Fix clearing kv cache between requests with the same prompt

* fix powershell script

stub generate outputs for lint

refactor tensor read

More logging for gpu management

Fix an ordering glitch of dlerr/dlclose and add more logging to help
root cause some crashes users are hitting. This also refines the
function pointer names to use the underlying function names instead
of simplified names for readability.

Report more information about GPUs in verbose mode

This adds additional calls to both CUDA and ROCm management libraries to
discover additional attributes about the GPU(s) detected in the system, and
wires up runtime verbosity selection.  When users hit problems with GPUs we can
ask them to run with `OLLAMA_DEBUG=1 ollama serve` and share the results.