GitLab

fix: pad tensor item if ge zero

this produces a nicer output since both positive and negative values
produces the same width

Better WantedBy declaration

The encoder cache needs to know the position of images in the input
stream so that it knows when to delete them. Previously images didn't
have a position, so we implied one by breaking batches before an
image and then assuming the image was in the first position. However,
multimodal objects are now given explicit positions in the input
stream, so we can use that instead.

Breaking batches was also a way to simulate a cross attention mask
for mllama. However, given that it only supports a single sequence
and a single image, this mask doesn't serve any real purpose.
Removing the batch break does not appear to affect the quality of
the output.

Most of this is simply moving the input data structures to a new
package to avoid import cycles.

It's ok to fail on startup but we shouldn't panic during runtime
based on user input. Downgrade the panic to a warning.

Similar to the llama engine, quantizing the KV cache requires
flash attention to be enabled through the Ollama server.

Backends can impose additional alignment requirements on buffer sizes.
We should ensure that we meet these or allocations can fail.

Models can disable causality for all or part of their processing
while continuing to store data in the KV cache.

Debug logging of every token has previously caused test timeouts
on slower machines.

this ensures the tensor is created on the right buffer type for backends
such as cpu

temporary until tensor loading can accurately account for vision models

some tensors should be created on specific backends to reduce number of
copies and improve performance

each cache layer creates and maintains its own context instead of using
a large context for all layers

some tensors are expected to be used in repeating layers but are not
themselves repeated. this change copies these tensors into the same
backends as their repeating counterparts to minimize copying tensors
between backends

use a similar strategy as llama.cpp for deciding where tensors should be
allocated. this will be improved later to be aware of usable memory
before assigning the tensor

This change bring in various interface cleanups along with greatly improving the performance of the sampler.

Tested with llama3.2 on local machine.
Improves performance from ~ 70 tokens/s -> 135 tokens/s with topK(40) enabled.
Without topK performance is ~ 110 tokens/s