Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
ModelZoo
ResNet50_tensorflow
Commits
733a752d
Commit
733a752d
authored
May 28, 2019
by
guptapriya
Browse files
Add shuffle to dataset records
This shuffling should help in getting shuffling each epoch.
parent
9b7b64be
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
4 additions
and
1 deletion
+4
-1
official/transformer/v2/data_pipeline.py
official/transformer/v2/data_pipeline.py
+4
-1
No files found.
official/transformer/v2/data_pipeline.py
View file @
733a752d
...
@@ -235,7 +235,10 @@ def _read_and_batch_from_files(
...
@@ -235,7 +235,10 @@ def _read_and_batch_from_files(
# Remove examples where the input or target length exceeds the maximum length,
# Remove examples where the input or target length exceeds the maximum length,
dataset
=
dataset
.
filter
(
lambda
x
,
y
:
_filter_max_length
((
x
,
y
),
max_length
))
dataset
=
dataset
.
filter
(
lambda
x
,
y
:
_filter_max_length
((
x
,
y
),
max_length
))
if
shuffle
:
dataset
=
dataset
.
shuffle
(
20000
)
if
static_batch
:
if
static_batch
:
dataset
=
dataset
.
padded_batch
(
dataset
=
dataset
.
padded_batch
(
batch_size
//
max_length
,
([
max_length
],
[
max_length
]),
batch_size
//
max_length
,
([
max_length
],
[
max_length
]),
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment