GitLab

Add CUDA Driver API for GPU discovery

Co-authored-by: Bruce MacDonald <brucewmacdonald@gmail.com>

* fix llava models not working after first request

* individual requests only for llava models

Centralize server config handling

This moves all the env var reading into one central module
and logs the loaded config once at startup which should
help in troubleshooting user server logs

---------
Co-authored-by: Patrick Devine <patrick@infrahq.com>

Make maximum pending request configurable

This also bumps up the default to be 50 queued requests
instead of 10.

Explain the 2 different windows download options

omit prompt and generate settings from final response

Fix lint warnings

rename parser to model/file

* Update 'llama2' -> 'llama3' in most places

---------
Co-authored-by: Patrick Devine <patrick@infrahq.com>

Soften timeouts on sched unit tests

This gives us more headroom on the scheduler tests to tamp
down some flakes.

refactor modelfile parser

fix line ending

replace CRLF with LF

* Add a /clear command

* change help messages

---------
Co-authored-by: Patrick Devine <patrick@infrahq.com>

server: destination invalid

types/model: fix name for hostport

Fix/issue 3736: When runners are closing or expiring. Scheduler is getting dirty VRAM size readings.