• Jerry Ma's avatar
    Add dataset class for weighted sampling with replacement. (#861) · a8a85c26
    Jerry Ma authored
    Summary:
    As discussed with Naman earlier today. Weighted sampling with
    replacement can be done on a per-epoch basis using `set_epoch()`
    functionality, which generates the samples as a function of random seed
    and epoch.
    
    Additionally, `FairseqTask` needs to set the starting epoch for the
    dataset at the very beginning of iterator construction.
    
    Not yet implemented is the per-epoch iterator construction, which
    is necessary to actually regenerate the batches for each epoch.
    Pull Request resolved: https://github.com/fairinternal/fairseq-py/pull/861
    
    Differential Revision: D17460687
    
    Pulled By: jma127
    
    fbshipit-source-id: 1c2a54f04ac96b3561c100a6fd66a9fccbe3c658
    a8a85c26
test_resampling_dataset.py 3.29 KB