Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
fairscale
Commits
cec011bb
Unverified
Commit
cec011bb
authored
Jun 14, 2021
by
anj-s
Committed by
GitHub
Jun 14, 2021
Browse files
[chore]Migrate away from legacy torchtext iterators (#713)
* migrate away from legacy iterators * fix lint error
parent
cbeda830
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
8 additions
and
1 deletion
+8
-1
benchmarks/datasets/wikitext2_data.py
benchmarks/datasets/wikitext2_data.py
+8
-1
No files found.
benchmarks/datasets/wikitext2_data.py
View file @
cec011bb
...
...
@@ -3,14 +3,21 @@
# This source code is licensed under the BSD license found in the
# LICENSE file in the root directory of this source tree.
from
distutils.version
import
LooseVersion
import
io
import
operator
import
tempfile
import
torch
from
torch.utils.data
import
DataLoader
import
torchtext
from
torchtext.data.utils
import
get_tokenizer
from
torchtext.utils
import
download_from_url
,
extract_archive
from
torchtext.vocab
import
build_vocab_from_iterator
if
operator
.
ge
(
torchtext
.
__version__
,
LooseVersion
(
"0.10.0"
)):
from
torchtext.legacy.vocab
import
build_vocab_from_iterator
else
:
from
torchtext.vocab
import
build_vocab_from_iterator
def
_batchify
(
data
,
batch_size
):
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment