pt-46528794.err

WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.738611 23192 ProcessGroupNCCL.cpp:835] [Rank 66] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.738819 23191 ProcessGroupNCCL.cpp:835] [Rank 65] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.740980 23190 ProcessGroupNCCL.cpp:835] [Rank 67] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.762174  1811 ProcessGroupNCCL.cpp:835] [Rank 29] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.764740  3052 ProcessGroupNCCL.cpp:835] [Rank 87] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.762187  1812 ProcessGroupNCCL.cpp:835] [Rank 31] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.764750  3053 ProcessGroupNCCL.cpp:835] [Rank 85] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.764799  3054 ProcessGroupNCCL.cpp:835] [Rank 84] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.762208  1813 ProcessGroupNCCL.cpp:835] [Rank 28] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.762266  1814 ProcessGroupNCCL.cpp:835] [Rank 30] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.747395  2780 ProcessGroupNCCL.cpp:835] [Rank 25] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.747427  2784 ProcessGroupNCCL.cpp:835] [Rank 24] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.747443  2783 ProcessGroupNCCL.cpp:835] [Rank 26] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.747448  2782 ProcessGroupNCCL.cpp:835] [Rank 27] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.780685  3051 ProcessGroupNCCL.cpp:835] [Rank 86] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.784382 31911 ProcessGroupNCCL.cpp:835] [Rank 79] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.785180 31910 ProcessGroupNCCL.cpp:835] [Rank 77] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.786365 31908 ProcessGroupNCCL.cpp:835] [Rank 78] NCCL watchdog thread started!
I1109 12:05:43.783609   907 ProcessGroupNCCL.cpp:669] [Rank 29] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.783619   909 ProcessGroupNCCL.cpp:669] [Rank 31] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.783638   906 ProcessGroupNCCL.cpp:669] [Rank 28] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.783648   908 ProcessGroupNCCL.cpp:669] [Rank 30] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.790745  7019 ProcessGroupNCCL.cpp:835] [Rank 92] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.787755 31909 ProcessGroupNCCL.cpp:835] [Rank 76] NCCL watchdog thread started!
I1109 12:05:43.776654  1829 ProcessGroupNCCL.cpp:669] [Rank 27] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.776679  1828 ProcessGroupNCCL.cpp:669] [Rank 26] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.776686  1826 ProcessGroupNCCL.cpp:669] [Rank 24] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.776700  1827 ProcessGroupNCCL.cpp:669] [Rank 25] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.795926 31187 ProcessGroupNCCL.cpp:669] [Rank 79] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.795939 31185 ProcessGroupNCCL.cpp:669] [Rank 77] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.795955 31186 ProcessGroupNCCL.cpp:669] [Rank 78] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.795971 31184 ProcessGroupNCCL.cpp:669] [Rank 76] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.799852  7017 ProcessGroupNCCL.cpp:835] [Rank 93] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.801586  7020 ProcessGroupNCCL.cpp:835] [Rank 94] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.811719 31384 ProcessGroupNCCL.cpp:835] [Rank 5] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.808806 26632 ProcessGroupNCCL.cpp:835] [Rank 82] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.812083 31383 ProcessGroupNCCL.cpp:835] [Rank 4] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.807608 31917 ProcessGroupNCCL.cpp:835] [Rank 61] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.809540 26630 ProcessGroupNCCL.cpp:835] [Rank 81] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.792016 16252 ProcessGroupNCCL.cpp:835] [Rank 2] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810068 26631 ProcessGroupNCCL.cpp:835] [Rank 83] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.798055 25435 ProcessGroupNCCL.cpp:835] [Rank 91] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.798069 25434 ProcessGroupNCCL.cpp:835] [Rank 89] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.798086 25436 ProcessGroupNCCL.cpp:835] [Rank 88] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.798103 25437 ProcessGroupNCCL.cpp:835] [Rank 90] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810381 26629 ProcessGroupNCCL.cpp:835] [Rank 80] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.766757 23189 ProcessGroupNCCL.cpp:835] [Rank 64] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.809396 31914 ProcessGroupNCCL.cpp:835] [Rank 62] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.809392 27283 ProcessGroupNCCL.cpp:835] [Rank 33] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.809792 30029 ProcessGroupNCCL.cpp:835] [Rank 49] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810344 30030 ProcessGroupNCCL.cpp:835] [Rank 51] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.809420 27285 ProcessGroupNCCL.cpp:835] [Rank 35] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810529 30032 ProcessGroupNCCL.cpp:835] [Rank 50] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.809432 27286 ProcessGroupNCCL.cpp:835] [Rank 34] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.796785 23899 ProcessGroupNCCL.cpp:835] [Rank 45] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810613 30031 ProcessGroupNCCL.cpp:835] [Rank 48] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.796801 23901 ProcessGroupNCCL.cpp:835] [Rank 46] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.796821 23900 ProcessGroupNCCL.cpp:835] [Rank 47] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.796831 23898 ProcessGroupNCCL.cpp:835] [Rank 44] NCCL watchdog thread started!
I1109 12:05:43.799458 24492 ProcessGroupNCCL.cpp:669] [Rank 90] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.799486 24491 ProcessGroupNCCL.cpp:669] [Rank 89] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810114 31915 ProcessGroupNCCL.cpp:835] [Rank 63] NCCL watchdog thread started!
I1109 12:05:43.799516 24490 ProcessGroupNCCL.cpp:669] [Rank 88] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.799510 24493 ProcessGroupNCCL.cpp:669] [Rank 91] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810142 27284 ProcessGroupNCCL.cpp:835] [Rank 32] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810518 31916 ProcessGroupNCCL.cpp:835] [Rank 60] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.806154 27872 ProcessGroupNCCL.cpp:835] [Rank 73] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.806176 27871 ProcessGroupNCCL.cpp:835] [Rank 75] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.806185 27873 ProcessGroupNCCL.cpp:835] [Rank 72] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.792865  1516 ProcessGroupNCCL.cpp:835] [Rank 13] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.806206 27874 ProcessGroupNCCL.cpp:835] [Rank 74] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810251 24060 ProcessGroupNCCL.cpp:835] [Rank 10] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810259 24061 ProcessGroupNCCL.cpp:835] [Rank 9] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810338 24058 ProcessGroupNCCL.cpp:835] [Rank 11] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810349 24059 ProcessGroupNCCL.cpp:835] [Rank 8] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810405 29034 ProcessGroupNCCL.cpp:835] [Rank 71] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810407 29033 ProcessGroupNCCL.cpp:835] [Rank 70] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810417 29035 ProcessGroupNCCL.cpp:835] [Rank 69] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.811352 31352 ProcessGroupNCCL.cpp:835] [Rank 43] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.810433 29036 ProcessGroupNCCL.cpp:835] [Rank 68] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.811365 31349 ProcessGroupNCCL.cpp:835] [Rank 42] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.811373 31351 ProcessGroupNCCL.cpp:835] [Rank 41] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.811383 31350 ProcessGroupNCCL.cpp:835] [Rank 40] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.811134 16727 ProcessGroupNCCL.cpp:835] [Rank 57] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.794016  1515 ProcessGroupNCCL.cpp:835] [Rank 14] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.795776 16250 ProcessGroupNCCL.cpp:835] [Rank 1] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.803664  7016 ProcessGroupNCCL.cpp:835] [Rank 95] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.792483 22750 ProcessGroupNCCL.cpp:835] [Rank 20] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.812626 12537 ProcessGroupNCCL.cpp:835] [Rank 36] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.794432 22749 ProcessGroupNCCL.cpp:835] [Rank 23] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.798696 22747 ProcessGroupNCCL.cpp:835] [Rank 22] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.805809 22748 ProcessGroupNCCL.cpp:835] [Rank 21] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.795354  1517 ProcessGroupNCCL.cpp:835] [Rank 15] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.813421 12538 ProcessGroupNCCL.cpp:835] [Rank 39] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.801244 17398 ProcessGroupNCCL.cpp:835] [Rank 55] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.813822 12535 ProcessGroupNCCL.cpp:835] [Rank 37] NCCL watchdog thread started!
I1109 12:05:43.809396 26996 ProcessGroupNCCL.cpp:669] [Rank 73] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.809410 26997 ProcessGroupNCCL.cpp:669] [Rank 74] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.809425 26998 ProcessGroupNCCL.cpp:669] [Rank 75] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.809437 26995 ProcessGroupNCCL.cpp:669] [Rank 72] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.818766 31382 ProcessGroupNCCL.cpp:835] [Rank 7] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.815119 12536 ProcessGroupNCCL.cpp:835] [Rank 38] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.803184 17399 ProcessGroupNCCL.cpp:835] [Rank 54] NCCL watchdog thread started!
I1109 12:05:43.815835 30379 ProcessGroupNCCL.cpp:669] [Rank 42] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.815008 28148 ProcessGroupNCCL.cpp:669] [Rank 70] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.815863 30380 ProcessGroupNCCL.cpp:669] [Rank 43] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.815024 28147 ProcessGroupNCCL.cpp:669] [Rank 69] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.815034 28145 ProcessGroupNCCL.cpp:669] [Rank 68] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.815855 30378 ProcessGroupNCCL.cpp:669] [Rank 41] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.815882 30377 ProcessGroupNCCL.cpp:669] [Rank 40] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.815049 28150 ProcessGroupNCCL.cpp:669] [Rank 71] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.797041  5585 ProcessGroupNCCL.cpp:835] [Rank 17] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.798377  5584 ProcessGroupNCCL.cpp:835] [Rank 19] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.816632 16726 ProcessGroupNCCL.cpp:835] [Rank 58] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.800619 16251 ProcessGroupNCCL.cpp:835] [Rank 0] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.805239 17396 ProcessGroupNCCL.cpp:835] [Rank 52] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.806950 17397 ProcessGroupNCCL.cpp:835] [Rank 53] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.803568 16249 ProcessGroupNCCL.cpp:835] [Rank 3] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.804039  5583 ProcessGroupNCCL.cpp:835] [Rank 16] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.804042  5586 ProcessGroupNCCL.cpp:835] [Rank 18] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.821933 16725 ProcessGroupNCCL.cpp:835] [Rank 56] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.804730  1514 ProcessGroupNCCL.cpp:835] [Rank 12] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.824316 16724 ProcessGroupNCCL.cpp:835] [Rank 59] NCCL watchdog thread started!
I1109 12:05:43.824918 23099 ProcessGroupNCCL.cpp:669] [Rank 8] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.824932 23101 ProcessGroupNCCL.cpp:669] [Rank 9] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.824942 23103 ProcessGroupNCCL.cpp:669] [Rank 11] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.824954 23102 ProcessGroupNCCL.cpp:669] [Rank 10] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.827046 21865 ProcessGroupNCCL.cpp:669] [Rank 22] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.827071 21866 ProcessGroupNCCL.cpp:669] [Rank 23] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.827090 21864 ProcessGroupNCCL.cpp:669] [Rank 21] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.827100 21863 ProcessGroupNCCL.cpp:669] [Rank 20] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1109 12:05:43.846418 31381 ProcessGroupNCCL.cpp:835] [Rank 6] NCCL watchdog thread started!
I1109 12:05:43.847890 30601 ProcessGroupNCCL.cpp:669] [Rank 6] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.847877 30600 ProcessGroupNCCL.cpp:669] [Rank 5] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.847914 30599 ProcessGroupNCCL.cpp:669] [Rank 4] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.847930 30602 ProcessGroupNCCL.cpp:669] [Rank 7] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.915059 31209 ProcessGroupNCCL.cpp:669] [Rank 62] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.915076 31206 ProcessGroupNCCL.cpp:669] [Rank 60] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.915088 31207 ProcessGroupNCCL.cpp:669] [Rank 61] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.915100 31210 ProcessGroupNCCL.cpp:669] [Rank 63] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.902462 23025 ProcessGroupNCCL.cpp:669] [Rank 46] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.902482 23024 ProcessGroupNCCL.cpp:669] [Rank 45] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.902496 23026 ProcessGroupNCCL.cpp:669] [Rank 47] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.902506 23023 ProcessGroupNCCL.cpp:669] [Rank 44] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.903659 16527 ProcessGroupNCCL.cpp:669] [Rank 53] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.903671 16526 ProcessGroupNCCL.cpp:669] [Rank 52] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.919378  6048 ProcessGroupNCCL.cpp:669] [Rank 94] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.903695 16528 ProcessGroupNCCL.cpp:669] [Rank 54] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.903695 16529 ProcessGroupNCCL.cpp:669] [Rank 55] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.919389  6046 ProcessGroupNCCL.cpp:669] [Rank 92] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.919406  6049 ProcessGroupNCCL.cpp:669] [Rank 95] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.919417  6047 ProcessGroupNCCL.cpp:669] [Rank 93] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.915283 16038 ProcessGroupNCCL.cpp:669] [Rank 58] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.915297 16037 ProcessGroupNCCL.cpp:669] [Rank 57] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.915309 16039 ProcessGroupNCCL.cpp:669] [Rank 59] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.915318 16035 ProcessGroupNCCL.cpp:669] [Rank 56] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.898797 22275 ProcessGroupNCCL.cpp:669] [Rank 64] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.898820 22277 ProcessGroupNCCL.cpp:669] [Rank 66] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.898936 22276 ProcessGroupNCCL.cpp:669] [Rank 65] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.899005 22278 ProcessGroupNCCL.cpp:669] [Rank 67] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.921502 29372 ProcessGroupNCCL.cpp:669] [Rank 50] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.921516 29373 ProcessGroupNCCL.cpp:669] [Rank 51] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.921533 29369 ProcessGroupNCCL.cpp:669] [Rank 48] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.918622 26608 ProcessGroupNCCL.cpp:669] [Rank 32] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.921542 29371 ProcessGroupNCCL.cpp:669] [Rank 49] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.918643 26609 ProcessGroupNCCL.cpp:669] [Rank 33] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.918658 26611 ProcessGroupNCCL.cpp:669] [Rank 35] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.918660 26610 ProcessGroupNCCL.cpp:669] [Rank 34] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.906291  4756 ProcessGroupNCCL.cpp:669] [Rank 16] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.924923 11608 ProcessGroupNCCL.cpp:669] [Rank 37] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.924940 11609 ProcessGroupNCCL.cpp:669] [Rank 38] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.924954 11607 ProcessGroupNCCL.cpp:669] [Rank 36] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.924971 11610 ProcessGroupNCCL.cpp:669] [Rank 39] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.924710  2109 ProcessGroupNCCL.cpp:669] [Rank 85] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.924746  2111 ProcessGroupNCCL.cpp:669] [Rank 87] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.924723  2107 ProcessGroupNCCL.cpp:669] [Rank 84] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.924754  2110 ProcessGroupNCCL.cpp:669] [Rank 86] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.909556  4760 ProcessGroupNCCL.cpp:669] [Rank 19] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.911442  4758 ProcessGroupNCCL.cpp:669] [Rank 17] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.913373  4759 ProcessGroupNCCL.cpp:669] [Rank 18] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:44.008632 25919 ProcessGroupNCCL.cpp:669] [Rank 82] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:44.008651 25917 ProcessGroupNCCL.cpp:669] [Rank 80] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:44.008661 25918 ProcessGroupNCCL.cpp:669] [Rank 81] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:44.008675 25920 ProcessGroupNCCL.cpp:669] [Rank 83] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.989769   629 ProcessGroupNCCL.cpp:669] [Rank 15] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.989904   627 ProcessGroupNCCL.cpp:669] [Rank 13] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.989931   628 ProcessGroupNCCL.cpp:669] [Rank 14] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.989943   626 ProcessGroupNCCL.cpp:669] [Rank 12] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.992367 15187 ProcessGroupNCCL.cpp:669] [Rank 1] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.992380 15188 ProcessGroupNCCL.cpp:669] [Rank 2] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.992398 15185 ProcessGroupNCCL.cpp:669] [Rank 0] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:05:43.992398 15189 ProcessGroupNCCL.cpp:669] [Rank 3] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
I1109 12:05:48.245900 15185 ProcessGroupNCCL.cpp:1274] NCCL_DEBUG: INFO

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.61s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.61s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.61s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.61s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.68s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.69s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.69s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.69s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.80s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.79s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.80s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.81s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.67s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.67s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.73s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.73s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.66s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.67s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.73s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.68s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.73s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.68s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.68s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.71s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.68s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.71s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.69s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.71s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.63s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.74s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.72s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.63s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.75s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.64s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.74s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.64s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.66s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.66s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.65s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.71s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.71s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.65s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.66s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.74s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.71s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.65s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.65s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.73s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.71s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.65s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.73s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.76s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.73s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.72s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.65s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.72s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.65s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.72s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.65s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.72s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.65s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.67s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.68s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.68s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.64s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.68s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.65s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.69s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.65s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.69s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.65s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.69s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.69s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.62s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.62s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.62s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.62s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.71s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.70s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.71s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.71s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.64s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.63s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.64s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.64s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.78s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.77s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.78s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.77s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.69s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.86s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:27<00:27, 27.97s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:28<00:28, 28.08s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:28<00:28, 28.12s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:28<00:28, 28.14s/it]
Loading checkpoint shards:  50%|█████     | 1/2 [00:28<00:28, 28.13s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.04s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.03s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.04s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.04s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 22.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 22.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 22.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 22.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.96s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.96s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.96s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.96s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.96s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.96s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.96s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.96s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.96s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.97s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.03s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 21.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 20.99s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.04s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:43<00:00, 22.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.04s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.06s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.04s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.04s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.00s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.03s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.05s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.05s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.06s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.03s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.03s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 20.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.01s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.01s/it]
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.02s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.06s/it]
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.06s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.11s/it]
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.13s/it]
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.08s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.14s/it]
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.

Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 21.06s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:44<00:00, 22.12s/it]
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors
I1109 12:07:08.279536   488 ProcessGroupNCCL.cpp:835] [Rank 42] NCCL watchdog thread started!
I1109 12:07:08.279489 30379 ProcessGroupNCCL.cpp:669] [Rank 42] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.281185 21864 ProcessGroupNCCL.cpp:669] [Rank 21] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.281291 24365 ProcessGroupNCCL.cpp:835] [Rank 21] NCCL watchdog thread started!
I1109 12:07:08.281317 24366 ProcessGroupNCCL.cpp:835] [Rank 20] NCCL watchdog thread started!
I1109 12:07:08.281262 21863 ProcessGroupNCCL.cpp:669] [Rank 20] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.286374  1827 ProcessGroupNCCL.cpp:669] [Rank 25] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.286486  4426 ProcessGroupNCCL.cpp:835] [Rank 25] NCCL watchdog thread started!
I1109 12:07:08.286490  4427 ProcessGroupNCCL.cpp:835] [Rank 24] NCCL watchdog thread started!
I1109 12:07:08.286442  1826 ProcessGroupNCCL.cpp:669] [Rank 24] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.279827   489 ProcessGroupNCCL.cpp:835] [Rank 43] NCCL watchdog thread started!
I1109 12:07:08.286686  4428 ProcessGroupNCCL.cpp:835] [Rank 26] NCCL watchdog thread started!
I1109 12:07:08.279753 30380 ProcessGroupNCCL.cpp:669] [Rank 43] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.281432 21865 ProcessGroupNCCL.cpp:669] [Rank 22] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.286625  1828 ProcessGroupNCCL.cpp:669] [Rank 26] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.279861 30378 ProcessGroupNCCL.cpp:669] [Rank 41] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.281545 24367 ProcessGroupNCCL.cpp:835] [Rank 22] NCCL watchdog thread started!
I1109 12:07:08.280004   490 ProcessGroupNCCL.cpp:835] [Rank 41] NCCL watchdog thread started!
I1109 12:07:08.280022   492 ProcessGroupNCCL.cpp:835] [Rank 40] NCCL watchdog thread started!
I1109 12:07:08.281631 21866 ProcessGroupNCCL.cpp:669] [Rank 23] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.279981 30377 ProcessGroupNCCL.cpp:669] [Rank 40] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.281689 24368 ProcessGroupNCCL.cpp:835] [Rank 23] NCCL watchdog thread started!
I1109 12:07:08.286829  1829 ProcessGroupNCCL.cpp:669] [Rank 27] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.280354 28148 ProcessGroupNCCL.cpp:669] [Rank 70] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.280426 30691 ProcessGroupNCCL.cpp:835] [Rank 70] NCCL watchdog thread started!
I1109 12:07:08.281636 31187 ProcessGroupNCCL.cpp:669] [Rank 79] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.281746   822 ProcessGroupNCCL.cpp:835] [Rank 79] NCCL watchdog thread started!
I1109 12:07:08.286958  4429 ProcessGroupNCCL.cpp:835] [Rank 27] NCCL watchdog thread started!
I1109 12:07:08.281715 31186 ProcessGroupNCCL.cpp:669] [Rank 78] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.281827   823 ProcessGroupNCCL.cpp:835] [Rank 78] NCCL watchdog thread started!
I1109 12:07:08.281765 31185 ProcessGroupNCCL.cpp:669] [Rank 77] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.281867   824 ProcessGroupNCCL.cpp:835] [Rank 77] NCCL watchdog thread started!
I1109 12:07:08.280580 28150 ProcessGroupNCCL.cpp:669] [Rank 71] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.280694 30692 ProcessGroupNCCL.cpp:835] [Rank 71] NCCL watchdog thread started!
I1109 12:07:08.288532  4758 ProcessGroupNCCL.cpp:669] [Rank 17] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.288658  7214 ProcessGroupNCCL.cpp:835] [Rank 17] NCCL watchdog thread started!
I1109 12:07:08.288678  7215 ProcessGroupNCCL.cpp:835] [Rank 16] NCCL watchdog thread started!
I1109 12:07:08.288657  4756 ProcessGroupNCCL.cpp:669] [Rank 16] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.280786 30693 ProcessGroupNCCL.cpp:835] [Rank 68] NCCL watchdog thread started!
I1109 12:07:08.282076 31184 ProcessGroupNCCL.cpp:669] [Rank 76] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.288738  4759 ProcessGroupNCCL.cpp:669] [Rank 18] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.288847  7216 ProcessGroupNCCL.cpp:835] [Rank 18] NCCL watchdog thread started!
I1109 12:07:08.282176   825 ProcessGroupNCCL.cpp:835] [Rank 76] NCCL watchdog thread started!
I1109 12:07:08.280437 11608 ProcessGroupNCCL.cpp:669] [Rank 37] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.280534 14296 ProcessGroupNCCL.cpp:835] [Rank 37] NCCL watchdog thread started!
I1109 12:07:08.280761 28145 ProcessGroupNCCL.cpp:669] [Rank 68] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.288854  4760 ProcessGroupNCCL.cpp:669] [Rank 19] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.288935  7217 ProcessGroupNCCL.cpp:835] [Rank 19] NCCL watchdog thread started!
I1109 12:07:08.288336 15188 ProcessGroupNCCL.cpp:669] [Rank 2] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.280875 28147 ProcessGroupNCCL.cpp:669] [Rank 69] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.280987 30694 ProcessGroupNCCL.cpp:835] [Rank 69] NCCL watchdog thread started!
I1109 12:07:08.288458 15189 ProcessGroupNCCL.cpp:669] [Rank 3] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.288558 17945 ProcessGroupNCCL.cpp:835] [Rank 3] NCCL watchdog thread started!
I1109 12:07:08.280670 11609 ProcessGroupNCCL.cpp:669] [Rank 38] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.288497 17944 ProcessGroupNCCL.cpp:835] [Rank 2] NCCL watchdog thread started!
I1109 12:07:08.280702 11610 ProcessGroupNCCL.cpp:669] [Rank 39] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.288678 17946 ProcessGroupNCCL.cpp:835] [Rank 1] NCCL watchdog thread started!
I1109 12:07:08.288620 15187 ProcessGroupNCCL.cpp:669] [Rank 1] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.280814 14298 ProcessGroupNCCL.cpp:835] [Rank 39] NCCL watchdog thread started!
I1109 12:07:08.280835 14297 ProcessGroupNCCL.cpp:835] [Rank 38] NCCL watchdog thread started!
I1109 12:07:08.280948 11607 ProcessGroupNCCL.cpp:669] [Rank 36] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.281042 14299 ProcessGroupNCCL.cpp:835] [Rank 36] NCCL watchdog thread started!
I1109 12:07:08.289314 15185 ProcessGroupNCCL.cpp:669] [Rank 0] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.287496 26998 ProcessGroupNCCL.cpp:669] [Rank 75] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.287528 29506 ProcessGroupNCCL.cpp:835] [Rank 75] NCCL watchdog thread started!
I1109 12:07:08.289446 17947 ProcessGroupNCCL.cpp:835] [Rank 0] NCCL watchdog thread started!
I1109 12:07:08.287611 29507 ProcessGroupNCCL.cpp:835] [Rank 72] NCCL watchdog thread started!
I1109 12:07:08.287566 26995 ProcessGroupNCCL.cpp:669] [Rank 72] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.287781 29508 ProcessGroupNCCL.cpp:835] [Rank 73] NCCL watchdog thread started!
I1109 12:07:08.287703 26996 ProcessGroupNCCL.cpp:669] [Rank 73] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.287869 26997 ProcessGroupNCCL.cpp:669] [Rank 74] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.287973 29509 ProcessGroupNCCL.cpp:835] [Rank 74] NCCL watchdog thread started!
I1109 12:07:08.287228  6049 ProcessGroupNCCL.cpp:669] [Rank 95] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.287282  6047 ProcessGroupNCCL.cpp:669] [Rank 93] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.287370  8972 ProcessGroupNCCL.cpp:835] [Rank 93] NCCL watchdog thread started!
I1109 12:07:08.287458  8973 ProcessGroupNCCL.cpp:835] [Rank 92] NCCL watchdog thread started!
I1109 12:07:08.287418  6046 ProcessGroupNCCL.cpp:669] [Rank 92] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.287357  8971 ProcessGroupNCCL.cpp:835] [Rank 95] NCCL watchdog thread started!
I1109 12:07:08.287482  8974 ProcessGroupNCCL.cpp:835] [Rank 94] NCCL watchdog thread started!
I1109 12:07:08.287478  6048 ProcessGroupNCCL.cpp:669] [Rank 94] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.294358 25919 ProcessGroupNCCL.cpp:669] [Rank 82] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.294483 28023 ProcessGroupNCCL.cpp:835] [Rank 82] NCCL watchdog thread started!
I1109 12:07:08.294492 28024 ProcessGroupNCCL.cpp:835] [Rank 81] NCCL watchdog thread started!
I1109 12:07:08.294421 25918 ProcessGroupNCCL.cpp:669] [Rank 81] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.294567 25920 ProcessGroupNCCL.cpp:669] [Rank 83] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.294677 28025 ProcessGroupNCCL.cpp:835] [Rank 83] NCCL watchdog thread started!
I1109 12:07:08.294735 28026 ProcessGroupNCCL.cpp:835] [Rank 80] NCCL watchdog thread started!
I1109 12:07:08.294695 25917 ProcessGroupNCCL.cpp:669] [Rank 80] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.338613  2109 ProcessGroupNCCL.cpp:669] [Rank 85] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.338709  4860 ProcessGroupNCCL.cpp:835] [Rank 85] NCCL watchdog thread started!
I1109 12:07:08.338763  4861 ProcessGroupNCCL.cpp:835] [Rank 84] NCCL watchdog thread started!
I1109 12:07:08.338722  2107 ProcessGroupNCCL.cpp:669] [Rank 84] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.338838  2111 ProcessGroupNCCL.cpp:669] [Rank 87] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.338961  4862 ProcessGroupNCCL.cpp:835] [Rank 87] NCCL watchdog thread started!
I1109 12:07:08.338989  4863 ProcessGroupNCCL.cpp:835] [Rank 86] NCCL watchdog thread started!
I1109 12:07:08.338980  2110 ProcessGroupNCCL.cpp:669] [Rank 86] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.354326 26610 ProcessGroupNCCL.cpp:669] [Rank 34] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.354342 26609 ProcessGroupNCCL.cpp:669] [Rank 33] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.354434 28676 ProcessGroupNCCL.cpp:835] [Rank 34] NCCL watchdog thread started!
I1109 12:07:08.354470 28677 ProcessGroupNCCL.cpp:835] [Rank 33] NCCL watchdog thread started!
I1109 12:07:08.354533 26611 ProcessGroupNCCL.cpp:669] [Rank 35] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.354655 28678 ProcessGroupNCCL.cpp:835] [Rank 35] NCCL watchdog thread started!
I1109 12:07:08.354575 26608 ProcessGroupNCCL.cpp:669] [Rank 32] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.354676 28679 ProcessGroupNCCL.cpp:835] [Rank 32] NCCL watchdog thread started!
I1109 12:07:08.358031 16037 ProcessGroupNCCL.cpp:669] [Rank 57] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.358090 16039 ProcessGroupNCCL.cpp:669] [Rank 59] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.358119 18120 ProcessGroupNCCL.cpp:835] [Rank 57] NCCL watchdog thread started!
I1109 12:07:08.358218 18121 ProcessGroupNCCL.cpp:835] [Rank 59] NCCL watchdog thread started!
I1109 12:07:08.358311 16038 ProcessGroupNCCL.cpp:669] [Rank 58] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.358459 18122 ProcessGroupNCCL.cpp:835] [Rank 58] NCCL watchdog thread started!
I1109 12:07:08.358538 16035 ProcessGroupNCCL.cpp:669] [Rank 56] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.358603 18123 ProcessGroupNCCL.cpp:835] [Rank 56] NCCL watchdog thread started!
I1109 12:07:08.364681 31209 ProcessGroupNCCL.cpp:669] [Rank 62] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.364761   914 ProcessGroupNCCL.cpp:835] [Rank 62] NCCL watchdog thread started!
I1109 12:07:08.364889  3549 ProcessGroupNCCL.cpp:835] [Rank 28] NCCL watchdog thread started!
I1109 12:07:08.364820   906 ProcessGroupNCCL.cpp:669] [Rank 28] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.364773 31210 ProcessGroupNCCL.cpp:669] [Rank 63] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.364806   907 ProcessGroupNCCL.cpp:669] [Rank 29] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.364924  3548 ProcessGroupNCCL.cpp:835] [Rank 29] NCCL watchdog thread started!
I1109 12:07:08.364878 31207 ProcessGroupNCCL.cpp:669] [Rank 61] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.364895   915 ProcessGroupNCCL.cpp:835] [Rank 63] NCCL watchdog thread started!
I1109 12:07:08.364919   908 ProcessGroupNCCL.cpp:669] [Rank 30] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.364993   916 ProcessGroupNCCL.cpp:835] [Rank 61] NCCL watchdog thread started!
I1109 12:07:08.365032  3550 ProcessGroupNCCL.cpp:835] [Rank 30] NCCL watchdog thread started!
I1109 12:07:08.365072   917 ProcessGroupNCCL.cpp:835] [Rank 60] NCCL watchdog thread started!
I1109 12:07:08.364976   909 ProcessGroupNCCL.cpp:669] [Rank 31] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.365075  3551 ProcessGroupNCCL.cpp:835] [Rank 31] NCCL watchdog thread started!
I1109 12:07:08.365000 31206 ProcessGroupNCCL.cpp:669] [Rank 60] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.369995 16528 ProcessGroupNCCL.cpp:669] [Rank 54] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.370108 19035 ProcessGroupNCCL.cpp:835] [Rank 54] NCCL watchdog thread started!
I1109 12:07:08.370074 16527 ProcessGroupNCCL.cpp:669] [Rank 53] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.370175 19036 ProcessGroupNCCL.cpp:835] [Rank 53] NCCL watchdog thread started!
I1109 12:07:08.370452 16526 ProcessGroupNCCL.cpp:669] [Rank 52] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.370487 19038 ProcessGroupNCCL.cpp:835] [Rank 52] NCCL watchdog thread started!
I1109 12:07:08.370424 16529 ProcessGroupNCCL.cpp:669] [Rank 55] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.370527 19037 ProcessGroupNCCL.cpp:835] [Rank 55] NCCL watchdog thread started!
I1109 12:07:08.371943 22276 ProcessGroupNCCL.cpp:669] [Rank 65] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.372046 24734 ProcessGroupNCCL.cpp:835] [Rank 65] NCCL watchdog thread started!
I1109 12:07:08.372171 24735 ProcessGroupNCCL.cpp:835] [Rank 64] NCCL watchdog thread started!
I1109 12:07:08.372108 22275 ProcessGroupNCCL.cpp:669] [Rank 64] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.372331 22278 ProcessGroupNCCL.cpp:669] [Rank 67] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.372453 24736 ProcessGroupNCCL.cpp:835] [Rank 67] NCCL watchdog thread started!
I1109 12:07:08.372433 22277 ProcessGroupNCCL.cpp:669] [Rank 66] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.372527 24737 ProcessGroupNCCL.cpp:835] [Rank 66] NCCL watchdog thread started!
I1109 12:07:08.368604   627 ProcessGroupNCCL.cpp:669] [Rank 13] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.368707  3581 ProcessGroupNCCL.cpp:835] [Rank 13] NCCL watchdog thread started!
I1109 12:07:08.368700  3582 ProcessGroupNCCL.cpp:835] [Rank 12] NCCL watchdog thread started!
I1109 12:07:08.368662   626 ProcessGroupNCCL.cpp:669] [Rank 12] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.368832   628 ProcessGroupNCCL.cpp:669] [Rank 14] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.368942  3583 ProcessGroupNCCL.cpp:835] [Rank 14] NCCL watchdog thread started!
I1109 12:07:08.368960  3584 ProcessGroupNCCL.cpp:835] [Rank 15] NCCL watchdog thread started!
I1109 12:07:08.368876   629 ProcessGroupNCCL.cpp:669] [Rank 15] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.374657 30600 ProcessGroupNCCL.cpp:669] [Rank 5] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.374732   300 ProcessGroupNCCL.cpp:835] [Rank 5] NCCL watchdog thread started!
I1109 12:07:08.374866 30601 ProcessGroupNCCL.cpp:669] [Rank 6] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.375032   301 ProcessGroupNCCL.cpp:835] [Rank 6] NCCL watchdog thread started!
I1109 12:07:08.375046   302 ProcessGroupNCCL.cpp:835] [Rank 4] NCCL watchdog thread started!
I1109 12:07:08.374991 30599 ProcessGroupNCCL.cpp:669] [Rank 4] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.375012 30602 ProcessGroupNCCL.cpp:669] [Rank 7] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.375111   303 ProcessGroupNCCL.cpp:835] [Rank 7] NCCL watchdog thread started!
I1109 12:07:08.379796 24492 ProcessGroupNCCL.cpp:669] [Rank 90] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.379880 27105 ProcessGroupNCCL.cpp:835] [Rank 90] NCCL watchdog thread started!
I1109 12:07:08.373006 25692 ProcessGroupNCCL.cpp:835] [Rank 8] NCCL watchdog thread started!
I1109 12:07:08.372959 23099 ProcessGroupNCCL.cpp:669] [Rank 8] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.380694 25564 ProcessGroupNCCL.cpp:835] [Rank 45] NCCL watchdog thread started!
I1109 12:07:08.380645 23024 ProcessGroupNCCL.cpp:669] [Rank 45] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.379999 24491 ProcessGroupNCCL.cpp:669] [Rank 89] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.379987 24490 ProcessGroupNCCL.cpp:669] [Rank 88] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.380123 27107 ProcessGroupNCCL.cpp:835] [Rank 89] NCCL watchdog thread started!
I1109 12:07:08.380102 27106 ProcessGroupNCCL.cpp:835] [Rank 88] NCCL watchdog thread started!
I1109 12:07:08.373227 23101 ProcessGroupNCCL.cpp:669] [Rank 9] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.373335 25693 ProcessGroupNCCL.cpp:835] [Rank 9] NCCL watchdog thread started!
I1109 12:07:08.380844 23023 ProcessGroupNCCL.cpp:669] [Rank 44] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.380939 25565 ProcessGroupNCCL.cpp:835] [Rank 44] NCCL watchdog thread started!
I1109 12:07:08.380192 24493 ProcessGroupNCCL.cpp:669] [Rank 91] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.380986 25566 ProcessGroupNCCL.cpp:835] [Rank 46] NCCL watchdog thread started!
I1109 12:07:08.380287 27108 ProcessGroupNCCL.cpp:835] [Rank 91] NCCL watchdog thread started!
I1109 12:07:08.380942 23026 ProcessGroupNCCL.cpp:669] [Rank 47] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.373502 25694 ProcessGroupNCCL.cpp:835] [Rank 11] NCCL watchdog thread started!
I1109 12:07:08.380899 23025 ProcessGroupNCCL.cpp:669] [Rank 46] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.373447 23103 ProcessGroupNCCL.cpp:669] [Rank 11] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.381052 25567 ProcessGroupNCCL.cpp:835] [Rank 47] NCCL watchdog thread started!
I1109 12:07:08.373584 23102 ProcessGroupNCCL.cpp:669] [Rank 10] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.373701 25695 ProcessGroupNCCL.cpp:835] [Rank 10] NCCL watchdog thread started!
I1109 12:07:08.383754 29371 ProcessGroupNCCL.cpp:669] [Rank 49] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.383862 31528 ProcessGroupNCCL.cpp:835] [Rank 49] NCCL watchdog thread started!
I1109 12:07:08.384107 29373 ProcessGroupNCCL.cpp:669] [Rank 51] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.384075 29372 ProcessGroupNCCL.cpp:669] [Rank 50] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
I1109 12:07:08.384225 31529 ProcessGroupNCCL.cpp:835] [Rank 50] NCCL watchdog thread started!
I1109 12:07:08.384222 31530 ProcessGroupNCCL.cpp:835] [Rank 51] NCCL watchdog thread started!
I1109 12:07:08.384248 31531 ProcessGroupNCCL.cpp:835] [Rank 48] NCCL watchdog thread started!
I1109 12:07:08.384155 29369 ProcessGroupNCCL.cpp:669] [Rank 48] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_DESYNC_DEBUG: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0

  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]
  0%|          | 0/420 [00:00<?, ?it/s]I1109 12:07:42.237160 19048 ProcessGroupNCCL.cpp:1274] NCCL_DEBUG: INFO

  0%|          | 1/420 [01:14<8:40:51, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:29, 74.53s/it]
  0%|          | 1/420 [01:14<8:40:50, 74.58s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:50, 74.58s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:49, 74.58s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:50, 74.58s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:53, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:49, 74.58s/it]
                                                 

                                                 
                                                 


                                                 
                                                 
                                                 
                                                 

                                                 

                                                 

  0%|          | 1/420 [01:14<8:40:51, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]



                                                 

                                                 

                                                 

                                                 

                                                 
                                                 

                                                 

                                                 
                                                 

                                                 

                                                 
                                                 

                                                 

                                                 
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:50, 74.58s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]




  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:50, 74.58s/it]
  0%|          | 1/420 [01:14<8:40:53, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:49, 74.58s/it]
  0%|          | 1/420 [01:14<8:40:50, 74.58s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:29, 74.53s/it]
                                                 
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 1/420 [01:14<8:40:49, 74.58s/it]
  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]

  0%|          | 1/420 [01:14<8:40:52, 74.59s/it]
  0%|          | 2/420 [02:12<7:31:06, 64.75s/it]
  0%|          | 2/420 [02:12<7:31:01, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:04, 64.75s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:04, 64.75s/it]
  0%|          | 2/420 [02:12<7:30:57, 64.73s/it]
  0%|          | 2/420 [02:12<7:31:04, 64.75s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.75s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:05, 64.75s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:06, 64.75s/it]
                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 
                                                 

                                                 

  0%|          | 2/420 [02:12<7:31:06, 64.75s/it]

                                                 

                                                 

                                                 

                                                 

  0%|          | 2/420 [02:12<7:31:01, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.75s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:30:57, 64.73s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:05, 64.75s/it]
  0%|          | 2/420 [02:12<7:31:04, 64.75s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:06, 64.75s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
                                                 

  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:02, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:04, 64.75s/it]
  0%|          | 2/420 [02:12<7:31:03, 64.74s/it]
  0%|          | 2/420 [02:12<7:31:04, 64.75s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:43, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:38, 61.53s/it]
  1%|          | 3/420 [03:10<7:07:43, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:45, 61.55s/it]
  1%|          | 3/420 [03:10<7:07:43, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:40, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:43, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:42, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:42, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:43, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:40, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:44, 61.55s/it]
  1%|          | 3/420 [03:10<7:07:42, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:43, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:40, 61.54s/it]
                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 
                                                 

                                                 
                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 

                                                 
  1%|          | 3/420 [03:10<7:07:43, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:43, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:44, 61.55s/it]
  1%|          | 3/420 [03:10<7:07:43, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:45, 61.55s/it]
  1%|          | 3/420 [03:10<7:07:43, 61.54s/it]



                                                 

                                                 

                                                 

  1%|          | 3/420 [03:10<7:07:38, 61.53s/it]
  1%|          | 3/420 [03:10<7:07:42, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:40, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:40, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:42, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:43, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:40, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:42, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:41, 61.54s/it]
  1%|          | 3/420 [03:10<7:07:43, 61.54s/it]slurmstepd: error: *** JOB 46528794 ON e07r1n18 CANCELLED AT 2023-11-09T12:10:47 ***