Skip to content

GitLab

Commit a96b8c5f authored by A. Unique TensorFlower's avatar A. Unique TensorFlower
Browse files

Add weight decay regex to Adafactor and expose weight decay params to its config. 

PiperOrigin-RevId: 448278992
parent c953c6a7
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 0 deletions
official/modeling/optimization/configs/optimizer_config.py
View file @ a96b8c5f
...@@ -311,3 +311,5 @@ class AdafactorConfig(BaseOptimizerConfig): ...@@ -311,3 +311,5 @@ class AdafactorConfig(BaseOptimizerConfig):
min_dim_size_to_factor: int = 128 min_dim_size_to_factor: int = 128
epsilon1: float = 1e-30 epsilon1: float = 1e-30
epsilon2: float = 1e-3 epsilon2: float = 1e-3
weight_decay: Optional[float] = None
include_in_weight_decay: Optional[str] = None
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment