GitLab

This library will give us the most reliable free VRAM reporting on windows
to enable concurrent model scheduling.

Still not complete, needs some refinement to our prediction to understand the
discrete GPUs available space so we can see how many layers fit in each one
since we can't split one layer across multiple GPUs we can't treat free space
as one logical block

Now that we call the GPU discovery routines many times to
update memory, this splits initial discovery from free memory
updating.

The amdgpu drivers free VRAM reporting omits some other apps, so leverage the
upstream DRM driver which keeps better tabs on things

This reverts commit 476fb8e8.

* fix oneapi errors on windows 10

Under stress scenarios we're seeing OOMs so this should help stabilize
the allocations under heavy concurrency stress.

The GPU drivers take a while to update their free memory reporting, so we need
to wait until the values converge with what we're expecting before proceeding
to start another runner in order to get an accurate picture.

This cleans up the logging for GPU discovery a bit, and can
serve as a foundation to report GPU information in a future UX.

Trying to live off the land for cuda libraries was not the right strategy.  We need to use the version we compiled against to ensure things work properly

This moves all the env var reading into one central module
and logs the loaded config once at startup which should
help in troubleshooting user server logs

For some reason this library gives incorrect GPU information, so skip it

We're seeing some corner cases with cudart which might be resolved by
switching to the driver API which comes bundled with the driver package

Correctly handle gfx90a discovery

Now that the llm runner is an executable and not just a dll, more users are facing
problems with security policy configurations on windows that prevent users
writing to directories and then executing binaries from the same location.
This change removes payloads from the main executable on windows and shifts them
over to be packaged in the installer and discovered based on the executables location.
This also adds a new zip file for people who want to "roll their own" installation model.

This change adds support for multiple concurrent requests, as well as
loading multiple models by spawning multiple runners. The default
settings are currently set at 1 concurrent request per model and only 1
loaded model at a time, but these can be adjusted by setting
OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS.

Leaving the cudart library loaded kept ~30m of memory
pinned in the GPU in the main process.  This change ensures
we don't hold GPU resources when idle.

We may have users that run into problems with our current
payload model, so this gives us an escape valve.

"cudart init failure: 35" isn't particularly helpful in the logs.

This should resolve a number of memory leak and stability defects by allowing
us to isolate llama.cpp in a separate process and shutdown when idle, and
gracefully restart if it has problems.  This also serves as a first step to be
able to run multiple copies to support multiple models concurrently.

count each layer independently when deciding gpu offloading