Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
gaoqiong
lm-evaluation-harness
Commits
ddc634f2
Commit
ddc634f2
authored
Jun 16, 2023
by
svenhendrikx
Browse files
Add logic to simple_evaluate to instantiate HFLM from transformers.PreTrainedModel instance
parent
42caa660
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
10 additions
and
1 deletion
+10
-1
lm_eval/evaluator.py
lm_eval/evaluator.py
+10
-1
No files found.
lm_eval/evaluator.py
View file @
ddc634f2
import
collections
import
collections
import
itertools
import
itertools
import
numpy
as
np
import
random
import
random
import
lm_eval.metrics
import
lm_eval.metrics
import
lm_eval.models
import
lm_eval.models
import
lm_eval.tasks
import
lm_eval.tasks
import
lm_eval.base
import
lm_eval.base
from
lm_eval.utils
import
positional_deprecated
,
run_task_tests
from
lm_eval.utils
import
positional_deprecated
,
run_task_tests
from
lm_eval.models.gpt2
import
HFLM
import
numpy
as
np
import
transformers
@
positional_deprecated
@
positional_deprecated
...
@@ -69,6 +73,11 @@ def simple_evaluate(
...
@@ -69,6 +73,11 @@ def simple_evaluate(
lm
=
lm_eval
.
models
.
get_model
(
model
).
create_from_arg_string
(
lm
=
lm_eval
.
models
.
get_model
(
model
).
create_from_arg_string
(
model_args
,
{
"batch_size"
:
batch_size
,
"device"
:
device
}
model_args
,
{
"batch_size"
:
batch_size
,
"device"
:
device
}
)
)
elif
isinstance
(
model
,
transformers
.
PreTrainedModel
):
lm
=
HFLM
(
pretrained
=
model
,
)
no_cache
=
True
else
:
else
:
assert
isinstance
(
model
,
lm_eval
.
base
.
LM
)
assert
isinstance
(
model
,
lm_eval
.
base
.
LM
)
lm
=
model
lm
=
model
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment