Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
ollama
Commits
3cf483fe
Commit
3cf483fe
authored
Apr 17, 2024
by
Michael Yang
Browse files
add stablelm graph calculation
parent
c8afe716
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
6 additions
and
1 deletion
+6
-1
llm/ggml.go
llm/ggml.go
+6
-0
llm/server.go
llm/server.go
+0
-1
No files found.
llm/ggml.go
View file @
3cf483fe
...
@@ -381,6 +381,12 @@ func (llm GGML) GraphSize(context, batch uint64) (partialOffload, fullOffload ui
...
@@ -381,6 +381,12 @@ func (llm GGML) GraphSize(context, batch uint64) (partialOffload, fullOffload ui
)
)
partialOffload
=
4
*
batch
*
(
2
*
embedding
+
vocab
)
+
embedding
*
vocab
*
105
/
128
partialOffload
=
4
*
batch
*
(
2
*
embedding
+
vocab
)
+
embedding
*
vocab
*
105
/
128
case
"stablelm"
:
fullOffload
=
4
*
batch
*
(
context
*
(
1
+
heads
)
+
3
*
embedding
+
2
)
partialOffload
=
max
(
4
*
batch
*
(
vocab
+
2
*
embedding
),
fullOffload
,
)
}
}
return
return
...
...
llm/server.go
View file @
3cf483fe
...
@@ -112,7 +112,6 @@ func NewLlamaServer(model string, adapters, projectors []string, opts api.Option
...
@@ -112,7 +112,6 @@ func NewLlamaServer(model string, adapters, projectors []string, opts api.Option
var
memoryLayerOutput
uint64
var
memoryLayerOutput
uint64
for
k
,
v
:=
range
layers
{
for
k
,
v
:=
range
layers
{
if
!
strings
.
HasPrefix
(
k
,
"blk."
)
{
if
!
strings
.
HasPrefix
(
k
,
"blk."
)
{
slog
.
Info
(
"aaa"
,
"name"
,
k
,
"size"
,
format
.
HumanBytes2
(
v
.
size
()))
memoryLayerOutput
+=
v
.
size
()
memoryLayerOutput
+=
v
.
size
()
}
}
}
}
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment