Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
AutoAWQ
Commits
9c1ab9ba
Commit
9c1ab9ba
authored
Aug 21, 2023
by
Jiaming Tang
Browse files
[Minor] Temporarily change calibration dataset URL
parent
b190df35
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
1 addition
and
1 deletion
+1
-1
awq/utils/calib_data.py
awq/utils/calib_data.py
+1
-1
No files found.
awq/utils/calib_data.py
View file @
9c1ab9ba
...
@@ -4,7 +4,7 @@ from datasets import load_dataset
...
@@ -4,7 +4,7 @@ from datasets import load_dataset
def
get_calib_dataset
(
data
=
"pileval"
,
tokenizer
=
None
,
n_samples
=
512
,
block_size
=
512
):
def
get_calib_dataset
(
data
=
"pileval"
,
tokenizer
=
None
,
n_samples
=
512
,
block_size
=
512
):
if
data
==
"pileval"
:
if
data
==
"pileval"
:
dataset
=
load_dataset
(
"
json"
,
data_files
=
"https://the-eye.eu/public/AI
/pile
/
val
.jsonl.zst
"
,
split
=
"
trai
n"
)
dataset
=
load_dataset
(
"
mit-han-lab
/pile
-
val
-backup
"
,
split
=
"
validatio
n"
)
else
:
else
:
raise
NotImplementedError
raise
NotImplementedError
dataset
=
dataset
.
shuffle
(
seed
=
42
)
dataset
=
dataset
.
shuffle
(
seed
=
42
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment