"scripts/vscode:/vscode.git/clone" did not exist on "04812de2170336a7423987fadb1d923923aaa9f4"
  • Ran Chen's avatar
    Save to tmp directory on non-chief workers in model_training_utils · 682d36b5
    Ran Chen authored
    In a multi worker set up saving is done on each worker. If they're saving to the same location, e.g. GCS, there will be conflicts. With this change we save to temporary directory on non-chief workers.
    
    Note that, there may be synchronization in saving that needs all workers to participate, so we cannot only save on one worker.
    
    PiperOrigin-RevId: 300141152
    682d36b5
model_training_utils.py 19.7 KB