Unverified Commit e314c19a authored by muzhi1991's avatar muzhi1991 Committed by GitHub
Browse files

fix bug for the log of RNG states are not properly loaded exception. (#15638)


Co-authored-by: default avatarmuz <muzhi1991@limuzhideMBP-2.lan>
parent 2e11a043
...@@ -1638,7 +1638,7 @@ class Trainer: ...@@ -1638,7 +1638,7 @@ class Trainer:
try: try:
torch.cuda.random.set_rng_state_all(checkpoint_rng_state["cuda"]) torch.cuda.random.set_rng_state_all(checkpoint_rng_state["cuda"])
except Exception as e: except Exception as e:
logger.infor( logger.info(
f"Didn't manage to set back the RNG states of the GPU because of the following error:\n {e}" f"Didn't manage to set back the RNG states of the GPU because of the following error:\n {e}"
"\nThis won't yield the same results as if the training had not been interrupted." "\nThis won't yield the same results as if the training had not been interrupted."
) )
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment