Skip to content

GitLab

Commit fb569880 authored by Michael Yang's avatar Michael Yang
Browse files

account for image projection in token count

parent d046bee7
Show whitespace changes
Inline Side-by-side
Showing with 4 additions and 0 deletions
server/routes.go
View file @ fb569880
...@@ -1273,6 +1273,10 @@ func trimmedPrompt(ctx context.Context, chat *ChatHistory, model *Model) (string ...@@ -1273,6 +1273,10 @@ func trimmedPrompt(ctx context.Context, chat *ChatHistory, model *Model) (string
promptsToAdd = append(promptsToAdd, promptInfo{vars: chat.Prompts[i], tokenLen: len(encodedTokens)}) promptsToAdd = append(promptsToAdd, promptInfo{vars: chat.Prompts[i], tokenLen: len(encodedTokens)})
images = append(images, chat.Prompts[i].Images...) images = append(images, chat.Prompts[i].Images...)
// clip has a projection dimension of 768
// TODO: use kv['clip.vision.projection_dim'] from projection instead
totalTokenLength += 768 * len(chat.Prompts[i].Images)
} }
// ensure the system prompt is included, if not already // ensure the system prompt is included, if not already
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment