Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
AutoAWQ
Commits
d832a218
Commit
d832a218
authored
Sep 15, 2023
by
Casper
Browse files
Move shuffling up for datasets loaded with load_dataset
parent
a9cef34b
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
5 additions
and
3 deletions
+5
-3
awq/utils/calib_data.py
awq/utils/calib_data.py
+5
-3
No files found.
awq/utils/calib_data.py
View file @
d832a218
from
typing
import
List
,
Union
import
torch
import
torch
import
logging
import
logging
from
typing
import
List
,
Union
from
datasets
import
load_dataset
from
datasets
import
load_dataset
def
get_calib_dataset
(
data
:
Union
[
str
,
List
[
str
]]
=
"pileval"
,
def
get_calib_dataset
(
data
:
Union
[
str
,
List
[
str
]]
=
"pileval"
,
...
@@ -11,14 +11,16 @@ def get_calib_dataset(data: Union[str, List[str]] = "pileval",
...
@@ -11,14 +11,16 @@ def get_calib_dataset(data: Union[str, List[str]] = "pileval",
dataset
=
load_dataset
(
"mit-han-lab/pile-val-backup"
,
split
=
"validation"
)
dataset
=
load_dataset
(
"mit-han-lab/pile-val-backup"
,
split
=
"validation"
)
else
:
else
:
dataset
=
load_dataset
(
data
,
split
=
split
)
dataset
=
load_dataset
(
data
,
split
=
split
)
dataset
=
dataset
.
shuffle
(
seed
=
42
)
elif
isinstance
(
data
,
list
):
elif
isinstance
(
data
,
list
):
dataset
=
[{
text_column
:
text
}
for
text
in
data
]
dataset
=
[{
text_column
:
text
}
for
text
in
data
]
else
:
else
:
raise
NotImplementedError
(
raise
NotImplementedError
(
"Either pass a string to a huggingface dataset or a list"
"Either pass a string to a huggingface dataset or a list"
"that is preprocessed with one sample of text per element."
)
"that is preprocessed with one sample of text per element."
)
dataset
=
dataset
.
shuffle
(
seed
=
42
)
samples
=
[]
samples
=
[]
n_run
=
0
n_run
=
0
for
data
in
dataset
:
for
data
in
dataset
:
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment