Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
cdf4cd70
Unverified
Commit
cdf4cd70
authored
Jul 10, 2020
by
Tomo Lazovich
Committed by
GitHub
Jul 10, 2020
Browse files
[squad] add version tag to squad cache (#5669)
parent
223084e4
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
4 additions
and
1 deletion
+4
-1
src/transformers/data/datasets/squad.py
src/transformers/data/datasets/squad.py
+4
-1
No files found.
src/transformers/data/datasets/squad.py
View file @
cdf4cd70
...
...
@@ -113,9 +113,12 @@ class SquadDataset(Dataset):
raise
KeyError
(
"mode is not a valid split name"
)
self
.
mode
=
mode
# Load data features from cache or dataset file
version_tag
=
"v2"
if
args
.
version_2_with_negative
else
"v1"
cached_features_file
=
os
.
path
.
join
(
cache_dir
if
cache_dir
is
not
None
else
args
.
data_dir
,
"cached_{}_{}_{}"
.
format
(
mode
.
value
,
tokenizer
.
__class__
.
__name__
,
str
(
args
.
max_seq_length
),),
"cached_{}_{}_{}_{}"
.
format
(
mode
.
value
,
tokenizer
.
__class__
.
__name__
,
str
(
args
.
max_seq_length
),
version_tag
,
),
)
# Make sure only the first process in distributed training processes the dataset,
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment