• Ma Zerun's avatar
    [WIP] continue PR #784 (#1221) · 357b0dfb
    Ma Zerun authored
    
    
    * Add gradient cumulative optimizer
    
    fixes #190
    
    * Update optimizer.py
    
    * Update optimizer.py
    
    * fix loss scale improperly in last equivalent_iter
    
    * Add `GradientCumulativeOptimizerHook` in `__init__.py`.
    
    * Add docstring of `GradientCumulativeOptimizerHook`.
    
    * Add type check, BN warning and resume warning. And fix typo, lint the
    code.
    
    * Add unit test
    
    * Update docstring example.
    
    * Change GradientCumulativeOptimizerHook `__init__` arguments.
    
    * Add GradientCumulativeOptimzierHook unit tests with IterBasedRunner.
    
    * Add GradientCumulativeFp16OptimizerHook.
    
    * Add unit tests of GradientCumulativeFp16OptimizerHook
    
    * Use '!=' instead of '>' to determine resume
    Co-authored-by: default avatarZhiyuan Chen <this@zyc.ai>
    357b0dfb
test_hooks.py 49.5 KB