Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
__init__.py 138 Bytes
Newer Older
Jared Casper's avatar
Initial commit of multiprocess preprocess and extracted copy of fairseq's indexed_dataset.  
Jared Casper committed
1
2 
from . import indexed_dataset
from .bert_tokenization import FullTokenizer as FullBertTokenizer
Jared Casper's avatar
Plumbing for new albert dataset, including mods to arguments for data loaders.  
Jared Casper committed
3 
from .albert_dataset import AlbertDataset