pt-45668270.err

WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.000419 11324 ProcessGroupNCCL.cpp:835] [Rank 19] NCCL watchdog thread started!
I1027 11:25:36.000407 10511 ProcessGroupNCCL.cpp:669] [Rank 19] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:35.994814 24977 ProcessGroupNCCL.cpp:835] [Rank 62] NCCL watchdog thread started!
I1027 11:25:35.994799 24232 ProcessGroupNCCL.cpp:669] [Rank 62] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.000789 26825 ProcessGroupNCCL.cpp:835] [Rank 22] NCCL watchdog thread started!
I1027 11:25:36.000782 26095 ProcessGroupNCCL.cpp:669] [Rank 22] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.003156 27045 ProcessGroupNCCL.cpp:835] [Rank 71] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.000727 11325 ProcessGroupNCCL.cpp:835] [Rank 18] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:35.998217 32364 ProcessGroupNCCL.cpp:835] [Rank 95] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.000821 26826 ProcessGroupNCCL.cpp:835] [Rank 23] NCCL watchdog thread started!
I1027 11:25:36.000818 26096 ProcessGroupNCCL.cpp:669] [Rank 23] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:36.003134 26319 ProcessGroupNCCL.cpp:669] [Rank 71] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:36.000705 10510 ProcessGroupNCCL.cpp:669] [Rank 18] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:35.998208 31587 ProcessGroupNCCL.cpp:669] [Rank 95] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.003333 27046 ProcessGroupNCCL.cpp:835] [Rank 70] NCCL watchdog thread started!
I1027 11:25:36.003324 26318 ProcessGroupNCCL.cpp:669] [Rank 70] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.006203  8400 ProcessGroupNCCL.cpp:835] [Rank 14] NCCL watchdog thread started!
I1027 11:25:36.006176  7503 ProcessGroupNCCL.cpp:669] [Rank 14] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.009217 32366 ProcessGroupNCCL.cpp:835] [Rank 94] NCCL watchdog thread started!
I1027 11:25:36.009191 31586 ProcessGroupNCCL.cpp:669] [Rank 94] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.012204  3415 ProcessGroupNCCL.cpp:835] [Rank 42] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.002801 21157 ProcessGroupNCCL.cpp:835] [Rank 91] NCCL watchdog thread started!
I1027 11:25:36.012202  2637 ProcessGroupNCCL.cpp:669] [Rank 42] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:36.002804 20260 ProcessGroupNCCL.cpp:669] [Rank 91] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.014935 24794 ProcessGroupNCCL.cpp:835] [Rank 30] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.012230  3416 ProcessGroupNCCL.cpp:835] [Rank 43] NCCL watchdog thread started!
I1027 11:25:36.012221  2638 ProcessGroupNCCL.cpp:669] [Rank 43] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:36.014866 23880 ProcessGroupNCCL.cpp:669] [Rank 30] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.018532 23508 ProcessGroupNCCL.cpp:835] [Rank 38] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.019204 14639 ProcessGroupNCCL.cpp:835] [Rank 57] NCCL watchdog thread started!
I1027 11:25:36.018523 22823 ProcessGroupNCCL.cpp:669] [Rank 38] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:36.019199 14036 ProcessGroupNCCL.cpp:669] [Rank 57] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.025336 21138 ProcessGroupNCCL.cpp:835] [Rank 79] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.018599 23509 ProcessGroupNCCL.cpp:835] [Rank 39] NCCL watchdog thread started!
I1027 11:25:36.018594 22824 ProcessGroupNCCL.cpp:669] [Rank 39] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.019300 14640 ProcessGroupNCCL.cpp:835] [Rank 56] NCCL watchdog thread started!
I1027 11:25:36.025269 20449 ProcessGroupNCCL.cpp:669] [Rank 79] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:36.019295 14034 ProcessGroupNCCL.cpp:669] [Rank 56] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.025368 21139 ProcessGroupNCCL.cpp:835] [Rank 78] NCCL watchdog thread started!
I1027 11:25:36.025368 20448 ProcessGroupNCCL.cpp:669] [Rank 78] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.035486 25002 ProcessGroupNCCL.cpp:835] [Rank 74] NCCL watchdog thread started!
I1027 11:25:36.035516 24084 ProcessGroupNCCL.cpp:669] [Rank 74] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.035588 25001 ProcessGroupNCCL.cpp:835] [Rank 75] NCCL watchdog thread started!
I1027 11:25:36.035571 24085 ProcessGroupNCCL.cpp:669] [Rank 75] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.032754  3251 ProcessGroupNCCL.cpp:835] [Rank 10] NCCL watchdog thread started!
I1027 11:25:36.032749  2343 ProcessGroupNCCL.cpp:669] [Rank 10] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.203402 10842 ProcessGroupNCCL.cpp:835] [Rank 86] NCCL watchdog thread started!
I1027 11:25:36.203394 10075 ProcessGroupNCCL.cpp:669] [Rank 86] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.196825  3253 ProcessGroupNCCL.cpp:835] [Rank 11] NCCL watchdog thread started!
I1027 11:25:36.196812  2344 ProcessGroupNCCL.cpp:669] [Rank 11] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.203168  8402 ProcessGroupNCCL.cpp:835] [Rank 15] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.202105 12499 ProcessGroupNCCL.cpp:835] [Rank 2] NCCL watchdog thread started!
I1027 11:25:36.202036 11396 ProcessGroupNCCL.cpp:669] [Rank 2] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.202195 12500 ProcessGroupNCCL.cpp:835] [Rank 3] NCCL watchdog thread started!
I1027 11:25:36.202183 11397 ProcessGroupNCCL.cpp:669] [Rank 3] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:36.203161  7504 ProcessGroupNCCL.cpp:669] [Rank 15] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.203104 32441 ProcessGroupNCCL.cpp:835] [Rank 67] NCCL watchdog thread started!
I1027 11:25:36.203104 31690 ProcessGroupNCCL.cpp:669] [Rank 67] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.196961 14643 ProcessGroupNCCL.cpp:835] [Rank 59] NCCL watchdog thread started!
I1027 11:25:36.196966 14038 ProcessGroupNCCL.cpp:669] [Rank 59] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.197103 14644 ProcessGroupNCCL.cpp:835] [Rank 58] NCCL watchdog thread started!
I1027 11:25:36.197136 14037 ProcessGroupNCCL.cpp:669] [Rank 58] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.204802 10844 ProcessGroupNCCL.cpp:835] [Rank 87] NCCL watchdog thread started!
I1027 11:25:36.204813 10076 ProcessGroupNCCL.cpp:669] [Rank 87] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.205030 24796 ProcessGroupNCCL.cpp:835] [Rank 31] NCCL watchdog thread started!
I1027 11:25:36.205034 23881 ProcessGroupNCCL.cpp:669] [Rank 31] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.202742 15657 ProcessGroupNCCL.cpp:835] [Rank 80] NCCL watchdog thread started!
I1027 11:25:36.202737 14727 ProcessGroupNCCL.cpp:669] [Rank 80] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.193079 21159 ProcessGroupNCCL.cpp:835] [Rank 90] NCCL watchdog thread started!
I1027 11:25:36.193073 20259 ProcessGroupNCCL.cpp:669] [Rank 90] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.196311 24979 ProcessGroupNCCL.cpp:835] [Rank 63] NCCL watchdog thread started!
I1027 11:25:36.196305 24233 ProcessGroupNCCL.cpp:669] [Rank 63] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.204185 32443 ProcessGroupNCCL.cpp:835] [Rank 66] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.203037 14731 ProcessGroupNCCL.cpp:669] [Rank 83] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:36.203060 15659 ProcessGroupNCCL.cpp:835] [Rank 83] NCCL watchdog thread started!
I1027 11:25:36.204182 31689 ProcessGroupNCCL.cpp:669] [Rank 66] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.203289 15658 ProcessGroupNCCL.cpp:835] [Rank 82] NCCL watchdog thread started!
I1027 11:25:36.203285 14730 ProcessGroupNCCL.cpp:669] [Rank 82] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.203330 15660 ProcessGroupNCCL.cpp:835] [Rank 81] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.202731 29614 ProcessGroupNCCL.cpp:835] [Rank 6] NCCL watchdog thread started!
I1027 11:25:36.203330 14729 ProcessGroupNCCL.cpp:669] [Rank 81] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:36.202728 28730 ProcessGroupNCCL.cpp:669] [Rank 6] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.202839 29615 ProcessGroupNCCL.cpp:835] [Rank 5] NCCL watchdog thread started!
I1027 11:25:36.202837 28729 ProcessGroupNCCL.cpp:669] [Rank 5] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.202972 29616 ProcessGroupNCCL.cpp:835] [Rank 4] NCCL watchdog thread started!
I1027 11:25:36.202955 28728 ProcessGroupNCCL.cpp:669] [Rank 4] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.203045 29617 ProcessGroupNCCL.cpp:835] [Rank 7] NCCL watchdog thread started!
I1027 11:25:36.203034 28731 ProcessGroupNCCL.cpp:669] [Rank 7] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.207926 16262 ProcessGroupNCCL.cpp:835] [Rank 35] NCCL watchdog thread started!
I1027 11:25:36.207923 15467 ProcessGroupNCCL.cpp:669] [Rank 35] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.207968 16263 ProcessGroupNCCL.cpp:835] [Rank 34] NCCL watchdog thread started!
I1027 11:25:36.207963 15466 ProcessGroupNCCL.cpp:669] [Rank 34] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.208457 20897 ProcessGroupNCCL.cpp:835] [Rank 27] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.208469 20898 ProcessGroupNCCL.cpp:835] [Rank 26] NCCL watchdog thread started!
I1027 11:25:36.208447 20095 ProcessGroupNCCL.cpp:669] [Rank 26] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:36.208431 20096 ProcessGroupNCCL.cpp:669] [Rank 27] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.211433 11294 ProcessGroupNCCL.cpp:835] [Rank 44] NCCL watchdog thread started!
I1027 11:25:36.211432 10427 ProcessGroupNCCL.cpp:669] [Rank 44] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.211643 11295 ProcessGroupNCCL.cpp:835] [Rank 46] NCCL watchdog thread started!
I1027 11:25:36.211635 10429 ProcessGroupNCCL.cpp:669] [Rank 46] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.211671 11296 ProcessGroupNCCL.cpp:835] [Rank 47] NCCL watchdog thread started!
I1027 11:25:36.211673 10430 ProcessGroupNCCL.cpp:669] [Rank 47] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.211741 11297 ProcessGroupNCCL.cpp:835] [Rank 45] NCCL watchdog thread started!
I1027 11:25:36.211752 10428 ProcessGroupNCCL.cpp:669] [Rank 45] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.210805 29328 ProcessGroupNCCL.cpp:669] [Rank 50] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:36.210826 29964 ProcessGroupNCCL.cpp:835] [Rank 50] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.210848 29965 ProcessGroupNCCL.cpp:835] [Rank 49] NCCL watchdog thread started!
I1027 11:25:36.210809 29327 ProcessGroupNCCL.cpp:669] [Rank 49] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.210841 29966 ProcessGroupNCCL.cpp:835] [Rank 51] NCCL watchdog thread started!
I1027 11:25:36.210840 29330 ProcessGroupNCCL.cpp:669] [Rank 51] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.211068 29967 ProcessGroupNCCL.cpp:835] [Rank 48] NCCL watchdog thread started!
I1027 11:25:36.211056 29325 ProcessGroupNCCL.cpp:669] [Rank 48] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.212285 22404 ProcessGroupNCCL.cpp:835] [Rank 52] NCCL watchdog thread started!
I1027 11:25:36.212198 21692 ProcessGroupNCCL.cpp:669] [Rank 52] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.212446 22405 ProcessGroupNCCL.cpp:835] [Rank 53] NCCL watchdog thread started!
I1027 11:25:36.212421 21694 ProcessGroupNCCL.cpp:669] [Rank 53] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.212572 22406 ProcessGroupNCCL.cpp:835] [Rank 54] NCCL watchdog thread started!
I1027 11:25:36.212548 21695 ProcessGroupNCCL.cpp:669] [Rank 54] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.212610 22407 ProcessGroupNCCL.cpp:835] [Rank 55] NCCL watchdog thread started!
I1027 11:25:36.212604 21696 ProcessGroupNCCL.cpp:669] [Rank 55] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.996578 32384 ProcessGroupNCCL.cpp:835] [Rank 93] NCCL watchdog thread started!
I1027 11:25:36.996528 31585 ProcessGroupNCCL.cpp:669] [Rank 93] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.996685 32385 ProcessGroupNCCL.cpp:835] [Rank 92] NCCL watchdog thread started!
I1027 11:25:36.996668 31583 ProcessGroupNCCL.cpp:669] [Rank 92] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.000815 21143 ProcessGroupNCCL.cpp:835] [Rank 76] NCCL watchdog thread started!
I1027 11:25:37.000811 20445 ProcessGroupNCCL.cpp:669] [Rank 76] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.000913 21144 ProcessGroupNCCL.cpp:835] [Rank 77] NCCL watchdog thread started!
I1027 11:25:37.000910 20447 ProcessGroupNCCL.cpp:669] [Rank 77] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.999774  3446 ProcessGroupNCCL.cpp:835] [Rank 41] NCCL watchdog thread started!
I1027 11:25:36.999768  2636 ProcessGroupNCCL.cpp:669] [Rank 41] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.999859  3447 ProcessGroupNCCL.cpp:835] [Rank 40] NCCL watchdog thread started!
I1027 11:25:36.999855  2634 ProcessGroupNCCL.cpp:669] [Rank 40] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.999356 26844 ProcessGroupNCCL.cpp:835] [Rank 20] NCCL watchdog thread started!
I1027 11:25:36.999341 26093 ProcessGroupNCCL.cpp:669] [Rank 20] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.999518 26845 ProcessGroupNCCL.cpp:835] [Rank 21] NCCL watchdog thread started!
I1027 11:25:36.999511 26094 ProcessGroupNCCL.cpp:669] [Rank 21] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.002300 10849 ProcessGroupNCCL.cpp:835] [Rank 84] NCCL watchdog thread started!
I1027 11:25:37.002291 10072 ProcessGroupNCCL.cpp:669] [Rank 84] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.999794 20901 ProcessGroupNCCL.cpp:835] [Rank 24] NCCL watchdog thread started!
I1027 11:25:36.999809 20092 ProcessGroupNCCL.cpp:669] [Rank 24] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.999497 11328 ProcessGroupNCCL.cpp:835] [Rank 16] NCCL watchdog thread started!
I1027 11:25:36.999490 10508 ProcessGroupNCCL.cpp:669] [Rank 16] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.001916 27074 ProcessGroupNCCL.cpp:835] [Rank 68] NCCL watchdog thread started!
I1027 11:25:37.001905 26315 ProcessGroupNCCL.cpp:669] [Rank 68] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.994484 23512 ProcessGroupNCCL.cpp:835] [Rank 37] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.001937  8405 ProcessGroupNCCL.cpp:835] [Rank 12] NCCL watchdog thread started!
I1027 11:25:36.994472 22822 ProcessGroupNCCL.cpp:669] [Rank 37] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:37.001932  7501 ProcessGroupNCCL.cpp:669] [Rank 12] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.002847 24817 ProcessGroupNCCL.cpp:835] [Rank 28] NCCL watchdog thread started!
I1027 11:25:37.002837 23877 ProcessGroupNCCL.cpp:669] [Rank 28] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.994053 24982 ProcessGroupNCCL.cpp:835] [Rank 60] NCCL watchdog thread started!
I1027 11:25:36.994030 24229 ProcessGroupNCCL.cpp:669] [Rank 60] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.995919  3256 ProcessGroupNCCL.cpp:835] [Rank 8] NCCL watchdog thread started!
I1027 11:25:36.995903  2340 ProcessGroupNCCL.cpp:669] [Rank 8] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.000293 20903 ProcessGroupNCCL.cpp:835] [Rank 25] NCCL watchdog thread started!
I1027 11:25:37.000286 20094 ProcessGroupNCCL.cpp:669] [Rank 25] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:36.991092 21162 ProcessGroupNCCL.cpp:835] [Rank 88] NCCL watchdog thread started!
I1027 11:25:36.991041 20256 ProcessGroupNCCL.cpp:669] [Rank 88] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.002975 10851 ProcessGroupNCCL.cpp:835] [Rank 85] NCCL watchdog thread started!
I1027 11:25:37.002985 10074 ProcessGroupNCCL.cpp:669] [Rank 85] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.002640  8407 ProcessGroupNCCL.cpp:835] [Rank 13] NCCL watchdog thread started!
I1027 11:25:37.002632  7502 ProcessGroupNCCL.cpp:669] [Rank 13] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.002166 32451 ProcessGroupNCCL.cpp:835] [Rank 64] NCCL watchdog thread started!
I1027 11:25:37.002157 31686 ProcessGroupNCCL.cpp:669] [Rank 64] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.201927 27076 ProcessGroupNCCL.cpp:835] [Rank 69] NCCL watchdog thread started!
I1027 11:25:37.201920 26317 ProcessGroupNCCL.cpp:669] [Rank 69] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.201634 12503 ProcessGroupNCCL.cpp:835] [Rank 1] NCCL watchdog thread started!
I1027 11:25:37.201622 11395 ProcessGroupNCCL.cpp:669] [Rank 1] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.200444 11330 ProcessGroupNCCL.cpp:835] [Rank 17] NCCL watchdog thread started!
I1027 11:25:37.200419 10509 ProcessGroupNCCL.cpp:669] [Rank 17] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.194712 24984 ProcessGroupNCCL.cpp:835] [Rank 61] NCCL watchdog thread started!
I1027 11:25:37.194701 24231 ProcessGroupNCCL.cpp:669] [Rank 61] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.195446 23514 ProcessGroupNCCL.cpp:835] [Rank 36] NCCL watchdog thread started!
I1027 11:25:37.195439 22820 ProcessGroupNCCL.cpp:669] [Rank 36] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.201769 16266 ProcessGroupNCCL.cpp:835] [Rank 33] NCCL watchdog thread started!
I1027 11:25:37.201746 15465 ProcessGroupNCCL.cpp:669] [Rank 33] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.197037  3258 ProcessGroupNCCL.cpp:835] [Rank 9] NCCL watchdog thread started!
I1027 11:25:37.197031  2342 ProcessGroupNCCL.cpp:669] [Rank 9] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.204128 25005 ProcessGroupNCCL.cpp:835] [Rank 72] NCCL watchdog thread started!
I1027 11:25:37.204124 24081 ProcessGroupNCCL.cpp:669] [Rank 72] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.202404 16268 ProcessGroupNCCL.cpp:835] [Rank 32] NCCL watchdog thread started!
I1027 11:25:37.202402 15463 ProcessGroupNCCL.cpp:669] [Rank 32] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.204619 24819 ProcessGroupNCCL.cpp:835] [Rank 29] NCCL watchdog thread started!
I1027 11:25:37.204551 23879 ProcessGroupNCCL.cpp:669] [Rank 29] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.204697 25007 ProcessGroupNCCL.cpp:835] [Rank 73] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.203425 32453 ProcessGroupNCCL.cpp:835] [Rank 65] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.192674 21164 ProcessGroupNCCL.cpp:835] [Rank 89] NCCL watchdog thread started!
I1027 11:25:37.203423 31688 ProcessGroupNCCL.cpp:669] [Rank 65] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:37.192667 20258 ProcessGroupNCCL.cpp:669] [Rank 89] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:25:37.204689 24083 ProcessGroupNCCL.cpp:669] [Rank 73] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1027 11:25:37.211710 12505 ProcessGroupNCCL.cpp:835] [Rank 0] NCCL watchdog thread started!
I1027 11:25:37.211706 11393 ProcessGroupNCCL.cpp:669] [Rank 0] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
pthread_mutex_timedlock() returned 110
Failed to initialize RSMI device mutex after 5 seconds. Previous execution may not have shutdown cleanly. To fix problem, stop all rocm_smi programs, and then delete the rocm_smi* shared memory files in /dev/shm.: Success
pthread_mutex_timedlock() returned 110
Failed to initialize RSMI device mutex after 5 seconds. Previous execution may not have shutdown cleanly. To fix problem, stop all rocm_smi programs, and then delete the rocm_smi* shared memory files in /dev/shm.: Success
pthread_mutex_timedlock() returned 110
Failed to initialize RSMI device mutex after 5 seconds. Previous execution may not have shutdown cleanly. To fix problem, stop all rocm_smi programs, and then delete the rocm_smi* shared memory files in /dev/shm.: Success
pthread_mutex_timedlock() returned 110
Failed to initialize RSMI device mutex after 5 seconds. Previous execution may not have shutdown cleanly. To fix problem, stop all rocm_smi programs, and then delete the rocm_smi* shared memory files in /dev/shm.: Success
rsmi_init() failed
rsmi_init() failed
rsmi_init() failed
rsmi_init() failed
I1027 11:25:45.175016 11393 ProcessGroupNCCL.cpp:1274] NCCL_DEBUG: INFO

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.14s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.14s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.14s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.14s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.18s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.26s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.14s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.18s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.27s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.14s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.19s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.17s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.14s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.18s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.14s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.19s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.25s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.25s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.20s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.25s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.20s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.20s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.20s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.21s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.21s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.30s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.28s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.29s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.29s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.23s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.22s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.20s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.20s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.23s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.18s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.17s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.20s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.18s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.17s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.18s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.17s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.24s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.16s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.17s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.24s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.16s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.14s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.15s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.23s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.16s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.14s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.23s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.14s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.14s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.19s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.19s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.19s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.19s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.20s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.31s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.34s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.33s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.34s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.32s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.37s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.28s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.31s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.26s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.26s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.26s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.26s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.30s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.30s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.30s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.30s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.34s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.46s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.47s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.58s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.62s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.72s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.03s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.03s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.03s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.13s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.13s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.13s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.12s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.10s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.11s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.10s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.11s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.03s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.03s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.03s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.11s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.11s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.11s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.11s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.04s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.04s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.12s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.13s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.05s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.05s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.05s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.05s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.08s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.08s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.08s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.08s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.08s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.09s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.08s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.08s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.08s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.08s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.06s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.10s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.06s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.10s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.06s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.10s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.06s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 19.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.04s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.10s/it]
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.05s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.05s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.05s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.12s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.12s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.12s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.12s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.06s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.17s/it]
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.08s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.08s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.15s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.15s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.15s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.14s/it]
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.16s/it]
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.05s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.18s/it]
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.22s/it]
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 20.14s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:42<00:00, 21.24s/it]
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
I1027 11:26:51.949496  7502 ProcessGroupNCCL.cpp:669] [Rank 13] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949605  9852 ProcessGroupNCCL.cpp:835] [Rank 13] NCCL watchdog thread started!
I1027 11:26:51.949599  7501 ProcessGroupNCCL.cpp:669] [Rank 12] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949707  9853 ProcessGroupNCCL.cpp:835] [Rank 12] NCCL watchdog thread started!
I1027 11:26:51.949003  1358 ProcessGroupNCCL.cpp:835] [Rank 66] NCCL watchdog thread started!
I1027 11:26:51.948947 31689 ProcessGroupNCCL.cpp:669] [Rank 66] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949792  9854 ProcessGroupNCCL.cpp:835] [Rank 14] NCCL watchdog thread started!
I1027 11:26:51.949082  1359 ProcessGroupNCCL.cpp:835] [Rank 65] NCCL watchdog thread started!
I1027 11:26:51.949018 31688 ProcessGroupNCCL.cpp:669] [Rank 65] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.947327 21695 ProcessGroupNCCL.cpp:669] [Rank 54] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.947432 23522 ProcessGroupNCCL.cpp:835] [Rank 54] NCCL watchdog thread started!
I1027 11:26:51.950186 20256 ProcessGroupNCCL.cpp:669] [Rank 88] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.950294 22552 ProcessGroupNCCL.cpp:835] [Rank 88] NCCL watchdog thread started!
I1027 11:26:51.949731  7503 ProcessGroupNCCL.cpp:669] [Rank 14] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949229 31690 ProcessGroupNCCL.cpp:669] [Rank 67] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.947433 21696 ProcessGroupNCCL.cpp:669] [Rank 55] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949116 20445 ProcessGroupNCCL.cpp:669] [Rank 76] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949229 22126 ProcessGroupNCCL.cpp:835] [Rank 76] NCCL watchdog thread started!
I1027 11:26:51.949371  1360 ProcessGroupNCCL.cpp:835] [Rank 67] NCCL watchdog thread started!
I1027 11:26:51.947487 21692 ProcessGroupNCCL.cpp:669] [Rank 52] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.950353 20260 ProcessGroupNCCL.cpp:669] [Rank 91] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949461  1361 ProcessGroupNCCL.cpp:835] [Rank 64] NCCL watchdog thread started!
I1027 11:26:51.947544 23523 ProcessGroupNCCL.cpp:835] [Rank 55] NCCL watchdog thread started!
I1027 11:26:51.950441 22553 ProcessGroupNCCL.cpp:835] [Rank 91] NCCL watchdog thread started!
I1027 11:26:51.949990  7504 ProcessGroupNCCL.cpp:669] [Rank 15] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949276 24231 ProcessGroupNCCL.cpp:669] [Rank 61] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949373 26315 ProcessGroupNCCL.cpp:835] [Rank 61] NCCL watchdog thread started!
I1027 11:26:51.949352 20449 ProcessGroupNCCL.cpp:669] [Rank 79] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949402 31686 ProcessGroupNCCL.cpp:669] [Rank 64] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.947618 23524 ProcessGroupNCCL.cpp:835] [Rank 52] NCCL watchdog thread started!
I1027 11:26:51.950549 22554 ProcessGroupNCCL.cpp:835] [Rank 89] NCCL watchdog thread started!
I1027 11:26:51.950112  9855 ProcessGroupNCCL.cpp:835] [Rank 15] NCCL watchdog thread started!
I1027 11:26:51.949306 24229 ProcessGroupNCCL.cpp:669] [Rank 60] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949427 26316 ProcessGroupNCCL.cpp:835] [Rank 60] NCCL watchdog thread started!
I1027 11:26:51.949481 22127 ProcessGroupNCCL.cpp:835] [Rank 79] NCCL watchdog thread started!
I1027 11:26:51.947643 21694 ProcessGroupNCCL.cpp:669] [Rank 53] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.947762 23525 ProcessGroupNCCL.cpp:835] [Rank 53] NCCL watchdog thread started!
I1027 11:26:51.950492 20258 ProcessGroupNCCL.cpp:669] [Rank 89] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949349 24233 ProcessGroupNCCL.cpp:669] [Rank 63] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949486 26317 ProcessGroupNCCL.cpp:835] [Rank 63] NCCL watchdog thread started!
I1027 11:26:51.949513 20448 ProcessGroupNCCL.cpp:669] [Rank 78] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.944944 14729 ProcessGroupNCCL.cpp:669] [Rank 81] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.945068 16997 ProcessGroupNCCL.cpp:835] [Rank 81] NCCL watchdog thread started!
I1027 11:26:51.947634 10508 ProcessGroupNCCL.cpp:669] [Rank 16] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.947746 12725 ProcessGroupNCCL.cpp:835] [Rank 16] NCCL watchdog thread started!
I1027 11:26:51.949496 26318 ProcessGroupNCCL.cpp:835] [Rank 62] NCCL watchdog thread started!
I1027 11:26:51.949404 24232 ProcessGroupNCCL.cpp:669] [Rank 62] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949640 22128 ProcessGroupNCCL.cpp:835] [Rank 78] NCCL watchdog thread started!
I1027 11:26:51.945032 14727 ProcessGroupNCCL.cpp:669] [Rank 80] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.947654 10509 ProcessGroupNCCL.cpp:669] [Rank 17] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.947773 12726 ProcessGroupNCCL.cpp:835] [Rank 17] NCCL watchdog thread started!
I1027 11:26:51.949669 20447 ProcessGroupNCCL.cpp:669] [Rank 77] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.945151 16998 ProcessGroupNCCL.cpp:835] [Rank 80] NCCL watchdog thread started!
I1027 11:26:51.947707 10511 ProcessGroupNCCL.cpp:669] [Rank 19] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949764 22129 ProcessGroupNCCL.cpp:835] [Rank 77] NCCL watchdog thread started!
I1027 11:26:51.945264 16999 ProcessGroupNCCL.cpp:835] [Rank 82] NCCL watchdog thread started!
I1027 11:26:51.947822 12727 ProcessGroupNCCL.cpp:835] [Rank 19] NCCL watchdog thread started!
I1027 11:26:51.945180 14730 ProcessGroupNCCL.cpp:669] [Rank 82] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.947847 10510 ProcessGroupNCCL.cpp:669] [Rank 18] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.945367 14731 ProcessGroupNCCL.cpp:669] [Rank 83] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.947969 12728 ProcessGroupNCCL.cpp:835] [Rank 18] NCCL watchdog thread started!
I1027 11:26:51.945502 17000 ProcessGroupNCCL.cpp:835] [Rank 83] NCCL watchdog thread started!
I1027 11:26:51.951151 20259 ProcessGroupNCCL.cpp:669] [Rank 90] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951236 22555 ProcessGroupNCCL.cpp:835] [Rank 90] NCCL watchdog thread started!
I1027 11:26:51.945559 26315 ProcessGroupNCCL.cpp:669] [Rank 68] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.945652 26319 ProcessGroupNCCL.cpp:669] [Rank 71] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.945685 28572 ProcessGroupNCCL.cpp:835] [Rank 68] NCCL watchdog thread started!
I1027 11:26:51.952735 23879 ProcessGroupNCCL.cpp:669] [Rank 29] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952816 26129 ProcessGroupNCCL.cpp:835] [Rank 29] NCCL watchdog thread started!
I1027 11:26:51.945760 26318 ProcessGroupNCCL.cpp:669] [Rank 70] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.945842 28574 ProcessGroupNCCL.cpp:835] [Rank 70] NCCL watchdog thread started!
I1027 11:26:51.945773 28573 ProcessGroupNCCL.cpp:835] [Rank 71] NCCL watchdog thread started!
I1027 11:26:51.945890 26317 ProcessGroupNCCL.cpp:669] [Rank 69] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.946013 28575 ProcessGroupNCCL.cpp:835] [Rank 69] NCCL watchdog thread started!
I1027 11:26:51.952797 23881 ProcessGroupNCCL.cpp:669] [Rank 31] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952850 23880 ProcessGroupNCCL.cpp:669] [Rank 30] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952908 26130 ProcessGroupNCCL.cpp:835] [Rank 31] NCCL watchdog thread started!
I1027 11:26:51.952955 26131 ProcessGroupNCCL.cpp:835] [Rank 30] NCCL watchdog thread started!
I1027 11:26:51.953125 26132 ProcessGroupNCCL.cpp:835] [Rank 28] NCCL watchdog thread started!
I1027 11:26:51.953084 23877 ProcessGroupNCCL.cpp:669] [Rank 28] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.950112 29328 ProcessGroupNCCL.cpp:669] [Rank 50] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.950232 31136 ProcessGroupNCCL.cpp:835] [Rank 50] NCCL watchdog thread started!
I1027 11:26:51.950434 29325 ProcessGroupNCCL.cpp:669] [Rank 48] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.950563 31137 ProcessGroupNCCL.cpp:835] [Rank 48] NCCL watchdog thread started!
I1027 11:26:51.950472 29327 ProcessGroupNCCL.cpp:669] [Rank 49] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.950568 31138 ProcessGroupNCCL.cpp:835] [Rank 49] NCCL watchdog thread started!
I1027 11:26:51.951445 10430 ProcessGroupNCCL.cpp:669] [Rank 47] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.950486 29330 ProcessGroupNCCL.cpp:669] [Rank 51] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.950613 31139 ProcessGroupNCCL.cpp:835] [Rank 51] NCCL watchdog thread started!
I1027 11:26:51.951589 12671 ProcessGroupNCCL.cpp:835] [Rank 47] NCCL watchdog thread started!
I1027 11:26:51.951678 10429 ProcessGroupNCCL.cpp:669] [Rank 46] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951462 15463 ProcessGroupNCCL.cpp:669] [Rank 32] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951588 17570 ProcessGroupNCCL.cpp:835] [Rank 32] NCCL watchdog thread started!
I1027 11:26:51.951716 10427 ProcessGroupNCCL.cpp:669] [Rank 44] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951812 12673 ProcessGroupNCCL.cpp:835] [Rank 44] NCCL watchdog thread started!
I1027 11:26:51.951828 12672 ProcessGroupNCCL.cpp:835] [Rank 46] NCCL watchdog thread started!
I1027 11:26:51.951530 15465 ProcessGroupNCCL.cpp:669] [Rank 33] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951676 17571 ProcessGroupNCCL.cpp:835] [Rank 33] NCCL watchdog thread started!
I1027 11:26:51.951695 17572 ProcessGroupNCCL.cpp:835] [Rank 34] NCCL watchdog thread started!
I1027 11:26:51.951634 15466 ProcessGroupNCCL.cpp:669] [Rank 34] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951701 15467 ProcessGroupNCCL.cpp:669] [Rank 35] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951804 17573 ProcessGroupNCCL.cpp:835] [Rank 35] NCCL watchdog thread started!
I1027 11:26:51.951908 10428 ProcessGroupNCCL.cpp:669] [Rank 45] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952037 12674 ProcessGroupNCCL.cpp:835] [Rank 45] NCCL watchdog thread started!
I1027 11:26:51.951092  4573 ProcessGroupNCCL.cpp:835] [Rank 10] NCCL watchdog thread started!
I1027 11:26:51.951079  2342 ProcessGroupNCCL.cpp:669] [Rank 9] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951072  2344 ProcessGroupNCCL.cpp:669] [Rank 11] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951015  2343 ProcessGroupNCCL.cpp:669] [Rank 10] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952960 22820 ProcessGroupNCCL.cpp:669] [Rank 36] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951189  4575 ProcessGroupNCCL.cpp:835] [Rank 9] NCCL watchdog thread started!
I1027 11:26:51.951203  4574 ProcessGroupNCCL.cpp:835] [Rank 11] NCCL watchdog thread started!
I1027 11:26:51.951225  2340 ProcessGroupNCCL.cpp:669] [Rank 8] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951320  4576 ProcessGroupNCCL.cpp:835] [Rank 8] NCCL watchdog thread started!
I1027 11:26:51.953114 22822 ProcessGroupNCCL.cpp:669] [Rank 37] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.953060 24875 ProcessGroupNCCL.cpp:835] [Rank 36] NCCL watchdog thread started!
I1027 11:26:51.953271 22823 ProcessGroupNCCL.cpp:669] [Rank 38] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.953379 24877 ProcessGroupNCCL.cpp:835] [Rank 38] NCCL watchdog thread started!
I1027 11:26:51.951431 28731 ProcessGroupNCCL.cpp:669] [Rank 7] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951532 31006 ProcessGroupNCCL.cpp:835] [Rank 7] NCCL watchdog thread started!
I1027 11:26:51.954536 24083 ProcessGroupNCCL.cpp:669] [Rank 73] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.954618 26340 ProcessGroupNCCL.cpp:835] [Rank 73] NCCL watchdog thread started!
I1027 11:26:51.954618 24081 ProcessGroupNCCL.cpp:669] [Rank 72] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.954713 26341 ProcessGroupNCCL.cpp:835] [Rank 72] NCCL watchdog thread started!
I1027 11:26:51.953224 24876 ProcessGroupNCCL.cpp:835] [Rank 37] NCCL watchdog thread started!
I1027 11:26:51.953326 22824 ProcessGroupNCCL.cpp:669] [Rank 39] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.953436 24878 ProcessGroupNCCL.cpp:835] [Rank 39] NCCL watchdog thread started!
I1027 11:26:51.948441 11393 ProcessGroupNCCL.cpp:669] [Rank 0] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.948542 16439 ProcessGroupNCCL.cpp:835] [Rank 0] NCCL watchdog thread started!
I1027 11:26:51.951676 28728 ProcessGroupNCCL.cpp:669] [Rank 4] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.954780 24084 ProcessGroupNCCL.cpp:669] [Rank 74] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951766 31007 ProcessGroupNCCL.cpp:835] [Rank 4] NCCL watchdog thread started!
I1027 11:26:51.951722 28729 ProcessGroupNCCL.cpp:669] [Rank 5] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.954936 26342 ProcessGroupNCCL.cpp:835] [Rank 74] NCCL watchdog thread started!
I1027 11:26:51.951814 31008 ProcessGroupNCCL.cpp:835] [Rank 5] NCCL watchdog thread started!
I1027 11:26:51.954890 24085 ProcessGroupNCCL.cpp:669] [Rank 75] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.948604 11397 ProcessGroupNCCL.cpp:669] [Rank 3] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.948693 16440 ProcessGroupNCCL.cpp:835] [Rank 3] NCCL watchdog thread started!
I1027 11:26:51.954980 26343 ProcessGroupNCCL.cpp:835] [Rank 75] NCCL watchdog thread started!
I1027 11:26:51.951839 28730 ProcessGroupNCCL.cpp:669] [Rank 6] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.951936 31009 ProcessGroupNCCL.cpp:835] [Rank 6] NCCL watchdog thread started!
I1027 11:26:51.951936 20096 ProcessGroupNCCL.cpp:669] [Rank 27] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952059 22487 ProcessGroupNCCL.cpp:835] [Rank 27] NCCL watchdog thread started!
I1027 11:26:51.953580  2638 ProcessGroupNCCL.cpp:669] [Rank 43] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952100 20092 ProcessGroupNCCL.cpp:669] [Rank 24] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.953667  2634 ProcessGroupNCCL.cpp:669] [Rank 40] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952216 22488 ProcessGroupNCCL.cpp:835] [Rank 24] NCCL watchdog thread started!
I1027 11:26:51.953668  4734 ProcessGroupNCCL.cpp:835] [Rank 43] NCCL watchdog thread started!
I1027 11:26:51.952178 20095 ProcessGroupNCCL.cpp:669] [Rank 26] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.953766  4735 ProcessGroupNCCL.cpp:835] [Rank 40] NCCL watchdog thread started!
I1027 11:26:51.952032 26093 ProcessGroupNCCL.cpp:669] [Rank 20] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952152 28321 ProcessGroupNCCL.cpp:835] [Rank 20] NCCL watchdog thread started!
I1027 11:26:51.952301 22489 ProcessGroupNCCL.cpp:835] [Rank 26] NCCL watchdog thread started!
I1027 11:26:51.953905  2637 ProcessGroupNCCL.cpp:669] [Rank 42] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952108 26095 ProcessGroupNCCL.cpp:669] [Rank 22] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949044 16442 ProcessGroupNCCL.cpp:835] [Rank 2] NCCL watchdog thread started!
I1027 11:26:51.948964 11396 ProcessGroupNCCL.cpp:669] [Rank 2] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952275 20094 ProcessGroupNCCL.cpp:669] [Rank 25] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.954031  4736 ProcessGroupNCCL.cpp:835] [Rank 42] NCCL watchdog thread started!
I1027 11:26:51.952210 28322 ProcessGroupNCCL.cpp:835] [Rank 22] NCCL watchdog thread started!
I1027 11:26:51.952401 22490 ProcessGroupNCCL.cpp:835] [Rank 25] NCCL watchdog thread started!
I1027 11:26:51.954025  2636 ProcessGroupNCCL.cpp:669] [Rank 41] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.949137 16443 ProcessGroupNCCL.cpp:835] [Rank 1] NCCL watchdog thread started!
I1027 11:26:51.949059 11395 ProcessGroupNCCL.cpp:669] [Rank 1] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.954125  4737 ProcessGroupNCCL.cpp:835] [Rank 41] NCCL watchdog thread started!
I1027 11:26:51.952327 26094 ProcessGroupNCCL.cpp:669] [Rank 21] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952345 26096 ProcessGroupNCCL.cpp:669] [Rank 23] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952450 28324 ProcessGroupNCCL.cpp:835] [Rank 23] NCCL watchdog thread started!
I1027 11:26:51.952440 28323 ProcessGroupNCCL.cpp:835] [Rank 21] NCCL watchdog thread started!
I1027 11:26:51.952198 14038 ProcessGroupNCCL.cpp:669] [Rank 59] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952296 14036 ProcessGroupNCCL.cpp:669] [Rank 57] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952248 14037 ProcessGroupNCCL.cpp:669] [Rank 58] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952311 15723 ProcessGroupNCCL.cpp:835] [Rank 59] NCCL watchdog thread started!
I1027 11:26:51.952401 15725 ProcessGroupNCCL.cpp:835] [Rank 57] NCCL watchdog thread started!
I1027 11:26:51.952426 14034 ProcessGroupNCCL.cpp:669] [Rank 56] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.952374 15724 ProcessGroupNCCL.cpp:835] [Rank 58] NCCL watchdog thread started!
I1027 11:26:51.952482 15726 ProcessGroupNCCL.cpp:835] [Rank 56] NCCL watchdog thread started!
I1027 11:26:51.955782 31583 ProcessGroupNCCL.cpp:669] [Rank 92] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.955814 31585 ProcessGroupNCCL.cpp:669] [Rank 93] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.955859 31587 ProcessGroupNCCL.cpp:669] [Rank 95] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.957005 10072 ProcessGroupNCCL.cpp:669] [Rank 84] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.957091 12176 ProcessGroupNCCL.cpp:835] [Rank 84] NCCL watchdog thread started!
I1027 11:26:51.955906  1173 ProcessGroupNCCL.cpp:835] [Rank 92] NCCL watchdog thread started!
I1027 11:26:51.957144 12177 ProcessGroupNCCL.cpp:835] [Rank 87] NCCL watchdog thread started!
I1027 11:26:51.955917  1174 ProcessGroupNCCL.cpp:835] [Rank 93] NCCL watchdog thread started!
I1027 11:26:51.955943  1175 ProcessGroupNCCL.cpp:835] [Rank 95] NCCL watchdog thread started!
I1027 11:26:51.956032 31586 ProcessGroupNCCL.cpp:669] [Rank 94] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.956175  1176 ProcessGroupNCCL.cpp:835] [Rank 94] NCCL watchdog thread started!
I1027 11:26:51.957084 10076 ProcessGroupNCCL.cpp:669] [Rank 87] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.957273 12178 ProcessGroupNCCL.cpp:835] [Rank 85] NCCL watchdog thread started!
I1027 11:26:51.957218 10074 ProcessGroupNCCL.cpp:669] [Rank 85] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1027 11:26:51.957412 12179 ProcessGroupNCCL.cpp:835] [Rank 86] NCCL watchdog thread started!
I1027 11:26:51.957370 10075 ProcessGroupNCCL.cpp:669] [Rank 86] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0

  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]I1027 11:27:34.182696 19423 ProcessGroupNCCL.cpp:1274] NCCL_DEBUG: INFO

  0%|          | 1/420 [01:03<7:25:16, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:13, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:14, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:15, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:12, 63.75s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:14, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:17, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:15, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:16, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:17, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:17, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:16, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:13, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:15, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:13, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:15, 63.76s/it]
                                                 

                                                 

                                                 

  0%|          | 1/420 [01:03<7:25:16, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
                                                 

                                                 

                                                 
                                                 

                                                 

                                                 

                                                 

                                                 
                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 
                                                 

                                                 
                                                 

                                                 

                                                 
                                                 

  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]





  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:17, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:12, 63.75s/it]
  0%|          | 1/420 [01:03<7:25:16, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:17, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:13, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:14, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:17, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:14, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:13, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:15, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:15, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:13, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:16, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:15, 63.76s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:18, 63.77s/it]
  0%|          | 1/420 [01:03<7:25:15, 63.76s/it]slurmstepd: error: *** JOB 45668270 ON e08r2n08 CANCELLED AT 2023-10-27T11:39:49 ***