GitLab

Fall back to CPU runner with zero layers

Integration fixes

Always use the sorted list of GPUs

Make sure the first GPU has the most free space

* rename `--quantization` to `--quantize`

* backwards

* Update api/types.go
Co-authored-by: Michael Yang <mxyng@pm.me>

---------
Co-authored-by: Michael Yang <mxyng@pm.me>

add phi2 mem

* dont clamp ctx size in `PredictServerFit`

* minimum 4 context

* remove context warning

Bump VRAM buffer back up

Under stress scenarios we're seeing OOMs so this should help stabilize
the allocations under heavy concurrency stress.

fix typo

only forward some env vars

log clean up

Fix race in shutdown logic

Ensure the runners are terminated

Wait for GPU free memory reporting to converge

The GPU drivers take a while to update their free memory reporting, so we need
to wait until the values converge with what we're expecting before proceeding
to start another runner in order to get an accurate picture.

Record more GPU information

This cleans up the logging for GPU discovery a bit, and can
serve as a foundation to report GPU information in a future UX.

Harden subprocess reaping

routes: skip invalid filepaths

Doc container usage and workaround for nvidia errors