• Jesse Gross's avatar
    ggml-backend: Don't recreate the scheduler for each context · e5bcc51a
    Jesse Gross authored
    We don't need to create and destroy the GGML scheduler for every
    context. This introduces extra CPU overhead for every forward
    pass and extra memory for contexts that don't actually get scheduled
    (for example, KV caches). We can instead just have one scheduler
    for the backend and reset it each time we call Compute.
    
    This improves token generation performance by 1-2% and removes
    scheduler create/destroy from profile traces.
    e5bcc51a
ggml.go 16.4 KB