"git@developer.sourcefind.cn:change/sglang.git" did not exist on "817d43705cd7f54aa5256e29edcd865c66250a5a"
Commit 79c1bad7 authored by Ruoxin Sang's avatar Ruoxin Sang Committed by A. Unique TensorFlower
Browse files

Don't early return if the global step is not updated by `num_steps` as...

Don't early return if the global step is not updated by `num_steps` as expected. This still allows summary and metric reporting.

PiperOrigin-RevId: 386967587
parent 29b4a322
...@@ -446,14 +446,13 @@ class Controller: ...@@ -446,14 +446,13 @@ class Controller:
f"{num_steps}. Old value was {current_step}, expected updated value " f"{num_steps}. Old value was {current_step}, expected updated value "
f"to be {expected_step}, but it was {self.global_step.numpy()}.") f"to be {expected_step}, but it was {self.global_step.numpy()}.")
logging.warning(message) logging.warning(message)
return
train_output = train_output or {} train_output = train_output or {}
for action in self.train_actions: for action in self.train_actions:
action(train_output) action(train_output)
train_output = tf.nest.map_structure(utils.get_value, train_output) train_output = tf.nest.map_structure(utils.get_value, train_output)
current_step = expected_step current_step = self.global_step.numpy()
steps_per_second = self.step_timer.steps_per_second() steps_per_second = self.step_timer.steps_per_second()
_log(f"train | step: {current_step: 6d} | " _log(f"train | step: {current_step: 6d} | "
f"steps/sec: {steps_per_second: 6.1f} | " f"steps/sec: {steps_per_second: 6.1f} | "
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment