GitLab

This reverts commit 163cd3e7.

Signed-off-by: Lei Jitang <leijitang@outlook.com>

* gpu: add env var for detecting intel oneapi gpus

* fix build error

Also removes an unused overall count variable

Signed-off-by: Lei Jitang <leijitang@outlook.com>

This should aid in troubleshooting by capturing and reporting the GPU
settings at startup in the logs along with all the other server settings.

Implement support for GPU env var workarounds, and leverage
this for the Vega RX 56 which needs
HSA_ENABLE_SDMA=0 set to work properly

This library will give us the most reliable free VRAM reporting on windows
to enable concurrent model scheduling.

Still not complete, needs some refinement to our prediction to understand the
discrete GPUs available space so we can see how many layers fit in each one
since we can't split one layer across multiple GPUs we can't treat free space
as one logical block

Now that we call the GPU discovery routines many times to
update memory, this splits initial discovery from free memory
updating.

The amdgpu drivers free VRAM reporting omits some other apps, so leverage the
upstream DRM driver which keeps better tabs on things

This reverts commit 476fb8e8.

* fix oneapi errors on windows 10

Under stress scenarios we're seeing OOMs so this should help stabilize
the allocations under heavy concurrency stress.

The GPU drivers take a while to update their free memory reporting, so we need
to wait until the values converge with what we're expecting before proceeding
to start another runner in order to get an accurate picture.

This cleans up the logging for GPU discovery a bit, and can
serve as a foundation to report GPU information in a future UX.

Trying to live off the land for cuda libraries was not the right strategy.  We need to use the version we compiled against to ensure things work properly

This moves all the env var reading into one central module
and logs the loaded config once at startup which should
help in troubleshooting user server logs

For some reason this library gives incorrect GPU information, so skip it

We're seeing some corner cases with cudart which might be resolved by
switching to the driver API which comes bundled with the driver package

Correctly handle gfx90a discovery

Now that the llm runner is an executable and not just a dll, more users are facing
problems with security policy configurations on windows that prevent users
writing to directories and then executing binaries from the same location.
This change removes payloads from the main executable on windows and shifts them
over to be packaged in the installer and discovered based on the executables location.
This also adds a new zip file for people who want to "roll their own" installation model.

This change adds support for multiple concurrent requests, as well as
loading multiple models by spawning multiple runners. The default
settings are currently set at 1 concurrent request per model and only 1
loaded model at a time, but these can be adjusted by setting
OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS.