GitLab

remove client.py

- prompt cache causes inferance to hang after some time

fix: request retry with error

improve cuda detection (rel. issue #1704)

Fix typo in arm mac arch script

Fix intel mac build

Make sure we're building an x86 ext_server lib when cross-compiling

remove double newlines in /set parameter

add max context length check

this fixes a subtle bug with makeRequestWithRetry where an HTTP status
error on a retried request will potentially not return the right err

Co-authored-by: Jeffrey Morgan <jmorganca@gmail.com>

After executing the `userdel ollama` command, I saw this message:

```sh
$ sudo userdel ollama
userdel: group ollama not removed because it has other members.
```

Which reminded me that I had to remove the dangling group too. For completeness, the uninstall instructions should do this too.

Thanks!

Fix up the CPU fallback selection

The memory changes and multi-variant change had some merge
glitches I missed.  This fixes them so we actually get the cpu llm lib
and best variant for the given system.

fix build and lint

Support multiple LLM libs; ROCm v5 and v6; Rosetta, AVX, and AVX2 compatible CPU builds

add lint and test on pull_request

This switches darwin to dynamic loading, and refactors the code now that no
static linking of the library is used on any platform

This reduces the built-in linux version to not use any vector extensions
which enables the resulting builds to run under Rosetta on MacOS in
Docker.  Then at runtime it checks for the actual CPU vector
extensions and loads the best CPU library available

In some cases we may want multiple variants for a given GPU type or CPU.
This adds logic to have an optional Variant which we can use to select
an optimal library, but also allows us to try multiple variants in case
some fail to load.

This can be useful for scenarios such as ROCm v5 vs v6 incompatibility
or potentially CPU features.

* increase minimum cuda overhead and fix minimum overhead for multi-gpu

* fix multi gpu overhead

* limit overhead to 10% of all gpus

* better wording

* allocate fixed amount before layers

* fixed only includes graph alloc