[Trainer] add error when passing `8bit`models (#20651)

* add error when passing `8bit`models * fix * improve message

[Trainer] add error when passing `8bit`models (#20651)
* add error when passing `8bit`models * fix * improve message
aac7b0d2 · Younes Belkada · GitHub · d151a8c5 · aac7b0d2
Unverified Commit aac7b0d2 authored Dec 07, 2022 by Younes Belkada Committed by GitHub Dec 07, 2022
Show whitespace changes
Inline Side-by-side

Showing with 7 additions and 0 deletions

src/transformers/trainer.py src/transformers/trainer.py +7 -0

No files found.
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -357,6 +357,13 @@ class Trainer:
        else:
            self.is_model_parallel = False

+        # At this stage the model is already loaded
+        if getattr(model, "is_loaded_in_8bit", False):
+            raise ValueError(
+                "The model you want to train is loaded in 8-bit precision. "
+                "Training an 8-bit model is not supported yet. "
+            )
+
        # Setup Sharded DDP training
        self.sharded_ddp = None
        if len(args.sharded_ddp) > 0: