• Jeff Rasley's avatar
    ZeRO-1 tune max-elems + bug fix (#532) · 08c96a1b
    Jeff Rasley authored
    * zero-1 memory fix
    
    * auto-tune max elems per comm to reduce padding/comm intervals
    
    * clean-up and added previously missing reduction options
    
    * fix testing backing to work with torch1.7
    08c96a1b
engine.py 60.5 KB