"...git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "4d4febb7aa3f49c29d7f4fc29bbc1760edd583b6"
Unverified Commit e314c19a authored by muzhi1991's avatar muzhi1991 Committed by GitHub
Browse files

fix bug for the log of RNG states are not properly loaded exception. (#15638)


Co-authored-by: default avatarmuz <muzhi1991@limuzhideMBP-2.lan>
parent 2e11a043
...@@ -1638,7 +1638,7 @@ class Trainer: ...@@ -1638,7 +1638,7 @@ class Trainer:
try: try:
torch.cuda.random.set_rng_state_all(checkpoint_rng_state["cuda"]) torch.cuda.random.set_rng_state_all(checkpoint_rng_state["cuda"])
except Exception as e: except Exception as e:
logger.infor( logger.info(
f"Didn't manage to set back the RNG states of the GPU because of the following error:\n {e}" f"Didn't manage to set back the RNG states of the GPU because of the following error:\n {e}"
"\nThis won't yield the same results as if the training had not been interrupted." "\nThis won't yield the same results as if the training had not been interrupted."
) )
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment