GitLab

count memory up to NumGPU if set by user

* Fixed the API endpoint /api/tags to return {models: []} instead of {models: null} when the model list is empty.

* Update server/routes.go

---------
Co-authored-by: Jeffrey Morgan <jmorganca@gmail.com>

Remove VRAM convergence check for windows

The APIs we query are optimistic on free space, and windows pages
VRAM, so we don't have to wait to see reported usage recover on unload

Update `LlamaScript` to point to new link from Legacy link.

removed inconsistent punctuation

* Correct typos.

* Correct typos.

Still used Legacy link.

This reverts commit 0fec3525.

use post token

Fix envconfig unit test

This reverts commit ce3b212d.

Fall back to CPU runner with zero layers

Integration fixes

Always use the sorted list of GPUs

Make sure the first GPU has the most free space

* rename `--quantization` to `--quantize`

* backwards

* Update api/types.go
Co-authored-by: Michael Yang <mxyng@pm.me>

---------
Co-authored-by: Michael Yang <mxyng@pm.me>

add phi2 mem

* dont clamp ctx size in `PredictServerFit`

* minimum 4 context

* remove context warning

Bump VRAM buffer back up

Under stress scenarios we're seeing OOMs so this should help stabilize
the allocations under heavy concurrency stress.