• Valentin's avatar
    Avoid looping when data exhausted (#14413) · a33168aa
    Valentin authored
    * stop training when a finite IterableDataset is exhausted
    
    when using an iterable dataset num_epochs is set to
    sys.maxsize to make sure all data is consumed
    likewise we want to set max_steps high enough
    but still stop when all data is consumed
    
    (cherry picked from commit 6f0e1d6363153da9051e93acffe1cbab3a3f3b12)
    
    * fix typo flase -> false
    
    * add test for stopping training on exhausted finite iterable dataset
    
    * remove redundant gradient_accumulation_steps
    
    * run make style
    
    reformat training_args docstring
    a33168aa
test_trainer.py 66.1 KB