Workaround for #27758 to avoid ZeroDivisionError (#28756)

c38a1227 · Traun Leyden · GitHub · 704b3f74 · c38a1227
Unverified Commit c38a1227 authored Mar 04, 2024 by Traun Leyden Committed by GitHub Mar 04, 2024
Hide whitespace changes
Inline Side-by-side

Showing with 2 additions and 1 deletion

src/transformers/trainer.py src/transformers/trainer.py +2 -1

No files found.
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -2080,7 +2080,8 @@ class Trainer:

        # add remaining tr_loss
        self._total_loss_scalar += tr_loss.item()
-        train_loss = self._total_loss_scalar / self.state.global_step
+        effective_global_step = max(self.state.global_step, 0.001)  # Avoid ZeroDivisionError
+        train_loss = self._total_loss_scalar / effective_global_step

        metrics = speed_metrics(
            "train",