Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
optimizers.rst 564 Bytes
Newer Older
Reza Yazdani's avatar
Add CPUAdam optimizer for zero-offload in deepspeed engine (#484)  
Reza Yazdani committed
1
2
3 
Optimizers
===================
Cheng Li's avatar
Add optimizers and schedules to RTD and updated the corresponding part in the website (#799)  
Cheng Li committed
4 
DeepSpeed offers high-performance implementations of ``Adam`` optimizer on CPU; ``FusedAdam``, ``FusedAdam``, ``OneBitAdam`` optimizers on GPU.
Reza Yazdani's avatar
Add CPUAdam optimizer for zero-offload in deepspeed engine (#484)  
Reza Yazdani committed
5
Cheng Li's avatar
Add optimizers and schedules to RTD and updated the corresponding part in the website (#799)  
Cheng Li committed
6 
Adam (CPU)
Reza Yazdani's avatar
Add CPUAdam optimizer for zero-offload in deepspeed engine (#484)  
Reza Yazdani committed
7
8
9 
----------------------------
.. autoclass:: deepspeed.ops.adam.DeepSpeedCPUAdam
Cheng Li's avatar
Add optimizers and schedules to RTD and updated the corresponding part in the website (#799)  
Cheng Li committed
10 
FusedAdam (GPU)
Reza Yazdani's avatar
Add CPUAdam optimizer for zero-offload in deepspeed engine (#484)  
Reza Yazdani committed
11 
----------------------------
Cheng Li's avatar
Add optimizers and schedules to RTD and updated the corresponding part in the website (#799)  
Cheng Li committed
12
13
14
15
16
17
18
19
20 
.. autoclass:: deepspeed.ops.adam.FusedAdam

FusedLamb (GPU)
----------------------------
.. autoclass:: deepspeed.ops.lamb.FusedLamb

OneBitAdam (GPU)
----------------------------
.. autoclass:: deepspeed.runtime.fp16.OneBitAdam