do not cache prompt (#2018)

- prompt cache causes inferance to hang after some time

do not cache prompt (#2018)
- prompt cache causes inferance to hang after some time
a897e833 · Bruce MacDonald · GitHub · eef50acc · a897e833
Unverified Commit a897e833 authored Jan 16, 2024 by Bruce MacDonald Committed by GitHub Jan 16, 2024
Show whitespace changes
Inline Side-by-side

Showing with 0 additions and 1 deletion

llm/dyn_ext_server.go llm/dyn_ext_server.go +0 -1

No files found.
--- a/llm/dyn_ext_server.go
+++ b/llm/dyn_ext_server.go
@@ -181,7 +181,6 @@ func (llm *dynExtServer) Predict(ctx context.Context, predict PredictOpts, fn fu
 		"seed":              predict.Options.Seed,
 		"stop":              predict.Options.Stop,
 		"image_data":        imageData,
-		"cache_prompt":      true,
 	}
 	if predict.Format == "json" {