GitLab

The recent refactoring of the memory prediction assumed all layers
are the same size, but for some models (like deepseek-coder-v2) this
is not the case, so our predictions were significantly off.

Tighten up memory prediction logging

Prior to this change, we logged the memory prediction multiple times
as the scheduler iterates to find a suitable configuration, which can be
confusing since only the last log before the server starts is actually valid.
This now logs once just before starting the server on the final configuration.
It also reports what library instead of always saying "offloading to gpu" when
using CPU.

Adjust mmap logic for cuda windows for faster model load

On Windows, recent llama.cpp changes make mmap slower in most
cases, so default to off.  This also implements a tri-state for
use_mmap so we can detect the difference between a user provided
value of true/false, or unspecified.

Revert powershell jobs, but keep nvcc and cmake parallelism

Implement custom github release action

nvcc supports parallelism (threads) and cmake + make can use -j,
while msbuild requires /p:CL_MPcount=8

This reverts commit 0577af98.

* llm: update llama.cpp submodule to `7c26775`

* disable `LLAMA_BLAS` for now

* `-DLLAMA_OPENMP=off`

Signed-off-by: Lei Jitang <leijitang@outlook.com>

* gpu: add env var for detecting intel oneapi gpus

* fix build error

Add some more debugging logs for intel discovery

Also removes an unused overall count variable

* Add Mod Time to Show

* Error Handling

* docs: add missing instruction for powershell build

The powershell script for building Ollama on Windows now requires the `ThreadJob` module. Add this to the instructions and dependency list.

* Update development.md

gpu: Fix build warning

This implements the release logic we want via gh cli
to support updating releases with rc tags in place and retain
release notes and other community reactions.

More parallelism on windows generate

Make the build faster

Rocm gfx900 workaround

Rocm v6 bump

Centralize GPU configuration vars

Signed-off-by: Lei Jitang <leijitang@outlook.com>

fix: "Skip searching for network devices"

This should aid in troubleshooting by capturing and reporting the GPU
settings at startup in the logs along with all the other server settings.

Implement support for GPU env var workarounds, and leverage
this for the Vega RX 56 which needs
HSA_ENABLE_SDMA=0 set to work properly

Enhanced GPU discovery and multi-gpu support with concurrency

While models are loading, the VRAM metrics are dynamic, so try
to load on a GPU that doesn't have a model actively loading, or wait
to avoid races that lead to OOMs

This library will give us the most reliable free VRAM reporting on windows
to enable concurrent model scheduling.

adjust timing on some tests so they don't timeout on small/slow GPUs

Our default behavior today is to try to fit into a single GPU if possible.
Some users would prefer the old behavior of always spreading across
multiple GPUs even if the model can fit into one.  This exposes that
tunable behavior.

Still not complete, needs some refinement to our prediction to understand the
discrete GPUs available space so we can see how many layers fit in each one
since we can't split one layer across multiple GPUs we can't treat free space
as one logical block