WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.738611 23192 ProcessGroupNCCL.cpp:835] [Rank 66] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.738819 23191 ProcessGroupNCCL.cpp:835] [Rank 65] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.740980 23190 ProcessGroupNCCL.cpp:835] [Rank 67] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.762174 1811 ProcessGroupNCCL.cpp:835] [Rank 29] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.764740 3052 ProcessGroupNCCL.cpp:835] [Rank 87] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.762187 1812 ProcessGroupNCCL.cpp:835] [Rank 31] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.764750 3053 ProcessGroupNCCL.cpp:835] [Rank 85] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.764799 3054 ProcessGroupNCCL.cpp:835] [Rank 84] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.762208 1813 ProcessGroupNCCL.cpp:835] [Rank 28] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.762266 1814 ProcessGroupNCCL.cpp:835] [Rank 30] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.747395 2780 ProcessGroupNCCL.cpp:835] [Rank 25] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.747427 2784 ProcessGroupNCCL.cpp:835] [Rank 24] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.747443 2783 ProcessGroupNCCL.cpp:835] [Rank 26] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.747448 2782 ProcessGroupNCCL.cpp:835] [Rank 27] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.780685 3051 ProcessGroupNCCL.cpp:835] [Rank 86] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.784382 31911 ProcessGroupNCCL.cpp:835] [Rank 79] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.785180 31910 ProcessGroupNCCL.cpp:835] [Rank 77] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.786365 31908 ProcessGroupNCCL.cpp:835] [Rank 78] NCCL watchdog thread started! I1109 12:05:43.783609 907 ProcessGroupNCCL.cpp:669] [Rank 29] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.783619 909 ProcessGroupNCCL.cpp:669] [Rank 31] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.783638 906 ProcessGroupNCCL.cpp:669] [Rank 28] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.783648 908 ProcessGroupNCCL.cpp:669] [Rank 30] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.790745 7019 ProcessGroupNCCL.cpp:835] [Rank 92] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.787755 31909 ProcessGroupNCCL.cpp:835] [Rank 76] NCCL watchdog thread started! I1109 12:05:43.776654 1829 ProcessGroupNCCL.cpp:669] [Rank 27] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.776679 1828 ProcessGroupNCCL.cpp:669] [Rank 26] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.776686 1826 ProcessGroupNCCL.cpp:669] [Rank 24] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.776700 1827 ProcessGroupNCCL.cpp:669] [Rank 25] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.795926 31187 ProcessGroupNCCL.cpp:669] [Rank 79] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.795939 31185 ProcessGroupNCCL.cpp:669] [Rank 77] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.795955 31186 ProcessGroupNCCL.cpp:669] [Rank 78] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.795971 31184 ProcessGroupNCCL.cpp:669] [Rank 76] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.799852 7017 ProcessGroupNCCL.cpp:835] [Rank 93] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.801586 7020 ProcessGroupNCCL.cpp:835] [Rank 94] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.811719 31384 ProcessGroupNCCL.cpp:835] [Rank 5] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.808806 26632 ProcessGroupNCCL.cpp:835] [Rank 82] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.812083 31383 ProcessGroupNCCL.cpp:835] [Rank 4] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.807608 31917 ProcessGroupNCCL.cpp:835] [Rank 61] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.809540 26630 ProcessGroupNCCL.cpp:835] [Rank 81] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.792016 16252 ProcessGroupNCCL.cpp:835] [Rank 2] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810068 26631 ProcessGroupNCCL.cpp:835] [Rank 83] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.798055 25435 ProcessGroupNCCL.cpp:835] [Rank 91] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.798069 25434 ProcessGroupNCCL.cpp:835] [Rank 89] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.798086 25436 ProcessGroupNCCL.cpp:835] [Rank 88] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.798103 25437 ProcessGroupNCCL.cpp:835] [Rank 90] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810381 26629 ProcessGroupNCCL.cpp:835] [Rank 80] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.766757 23189 ProcessGroupNCCL.cpp:835] [Rank 64] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.809396 31914 ProcessGroupNCCL.cpp:835] [Rank 62] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.809392 27283 ProcessGroupNCCL.cpp:835] [Rank 33] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.809792 30029 ProcessGroupNCCL.cpp:835] [Rank 49] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810344 30030 ProcessGroupNCCL.cpp:835] [Rank 51] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.809420 27285 ProcessGroupNCCL.cpp:835] [Rank 35] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810529 30032 ProcessGroupNCCL.cpp:835] [Rank 50] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.809432 27286 ProcessGroupNCCL.cpp:835] [Rank 34] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.796785 23899 ProcessGroupNCCL.cpp:835] [Rank 45] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810613 30031 ProcessGroupNCCL.cpp:835] [Rank 48] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.796801 23901 ProcessGroupNCCL.cpp:835] [Rank 46] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.796821 23900 ProcessGroupNCCL.cpp:835] [Rank 47] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.796831 23898 ProcessGroupNCCL.cpp:835] [Rank 44] NCCL watchdog thread started! I1109 12:05:43.799458 24492 ProcessGroupNCCL.cpp:669] [Rank 90] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.799486 24491 ProcessGroupNCCL.cpp:669] [Rank 89] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810114 31915 ProcessGroupNCCL.cpp:835] [Rank 63] NCCL watchdog thread started! I1109 12:05:43.799516 24490 ProcessGroupNCCL.cpp:669] [Rank 88] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.799510 24493 ProcessGroupNCCL.cpp:669] [Rank 91] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810142 27284 ProcessGroupNCCL.cpp:835] [Rank 32] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810518 31916 ProcessGroupNCCL.cpp:835] [Rank 60] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.806154 27872 ProcessGroupNCCL.cpp:835] [Rank 73] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.806176 27871 ProcessGroupNCCL.cpp:835] [Rank 75] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.806185 27873 ProcessGroupNCCL.cpp:835] [Rank 72] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.792865 1516 ProcessGroupNCCL.cpp:835] [Rank 13] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.806206 27874 ProcessGroupNCCL.cpp:835] [Rank 74] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810251 24060 ProcessGroupNCCL.cpp:835] [Rank 10] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810259 24061 ProcessGroupNCCL.cpp:835] [Rank 9] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810338 24058 ProcessGroupNCCL.cpp:835] [Rank 11] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810349 24059 ProcessGroupNCCL.cpp:835] [Rank 8] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810405 29034 ProcessGroupNCCL.cpp:835] [Rank 71] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810407 29033 ProcessGroupNCCL.cpp:835] [Rank 70] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810417 29035 ProcessGroupNCCL.cpp:835] [Rank 69] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.811352 31352 ProcessGroupNCCL.cpp:835] [Rank 43] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.810433 29036 ProcessGroupNCCL.cpp:835] [Rank 68] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.811365 31349 ProcessGroupNCCL.cpp:835] [Rank 42] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.811373 31351 ProcessGroupNCCL.cpp:835] [Rank 41] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.811383 31350 ProcessGroupNCCL.cpp:835] [Rank 40] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.811134 16727 ProcessGroupNCCL.cpp:835] [Rank 57] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.794016 1515 ProcessGroupNCCL.cpp:835] [Rank 14] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.795776 16250 ProcessGroupNCCL.cpp:835] [Rank 1] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.803664 7016 ProcessGroupNCCL.cpp:835] [Rank 95] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.792483 22750 ProcessGroupNCCL.cpp:835] [Rank 20] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.812626 12537 ProcessGroupNCCL.cpp:835] [Rank 36] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.794432 22749 ProcessGroupNCCL.cpp:835] [Rank 23] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.798696 22747 ProcessGroupNCCL.cpp:835] [Rank 22] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.805809 22748 ProcessGroupNCCL.cpp:835] [Rank 21] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.795354 1517 ProcessGroupNCCL.cpp:835] [Rank 15] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.813421 12538 ProcessGroupNCCL.cpp:835] [Rank 39] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.801244 17398 ProcessGroupNCCL.cpp:835] [Rank 55] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.813822 12535 ProcessGroupNCCL.cpp:835] [Rank 37] NCCL watchdog thread started! I1109 12:05:43.809396 26996 ProcessGroupNCCL.cpp:669] [Rank 73] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.809410 26997 ProcessGroupNCCL.cpp:669] [Rank 74] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.809425 26998 ProcessGroupNCCL.cpp:669] [Rank 75] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.809437 26995 ProcessGroupNCCL.cpp:669] [Rank 72] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.818766 31382 ProcessGroupNCCL.cpp:835] [Rank 7] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.815119 12536 ProcessGroupNCCL.cpp:835] [Rank 38] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.803184 17399 ProcessGroupNCCL.cpp:835] [Rank 54] NCCL watchdog thread started! I1109 12:05:43.815835 30379 ProcessGroupNCCL.cpp:669] [Rank 42] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.815008 28148 ProcessGroupNCCL.cpp:669] [Rank 70] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.815863 30380 ProcessGroupNCCL.cpp:669] [Rank 43] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.815024 28147 ProcessGroupNCCL.cpp:669] [Rank 69] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.815034 28145 ProcessGroupNCCL.cpp:669] [Rank 68] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.815855 30378 ProcessGroupNCCL.cpp:669] [Rank 41] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.815882 30377 ProcessGroupNCCL.cpp:669] [Rank 40] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.815049 28150 ProcessGroupNCCL.cpp:669] [Rank 71] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.797041 5585 ProcessGroupNCCL.cpp:835] [Rank 17] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.798377 5584 ProcessGroupNCCL.cpp:835] [Rank 19] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.816632 16726 ProcessGroupNCCL.cpp:835] [Rank 58] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.800619 16251 ProcessGroupNCCL.cpp:835] [Rank 0] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.805239 17396 ProcessGroupNCCL.cpp:835] [Rank 52] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.806950 17397 ProcessGroupNCCL.cpp:835] [Rank 53] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.803568 16249 ProcessGroupNCCL.cpp:835] [Rank 3] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.804039 5583 ProcessGroupNCCL.cpp:835] [Rank 16] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.804042 5586 ProcessGroupNCCL.cpp:835] [Rank 18] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.821933 16725 ProcessGroupNCCL.cpp:835] [Rank 56] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.804730 1514 ProcessGroupNCCL.cpp:835] [Rank 12] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.824316 16724 ProcessGroupNCCL.cpp:835] [Rank 59] NCCL watchdog thread started! I1109 12:05:43.824918 23099 ProcessGroupNCCL.cpp:669] [Rank 8] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.824932 23101 ProcessGroupNCCL.cpp:669] [Rank 9] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.824942 23103 ProcessGroupNCCL.cpp:669] [Rank 11] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.824954 23102 ProcessGroupNCCL.cpp:669] [Rank 10] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.827046 21865 ProcessGroupNCCL.cpp:669] [Rank 22] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.827071 21866 ProcessGroupNCCL.cpp:669] [Rank 23] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.827090 21864 ProcessGroupNCCL.cpp:669] [Rank 21] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.827100 21863 ProcessGroupNCCL.cpp:669] [Rank 20] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1109 12:05:43.846418 31381 ProcessGroupNCCL.cpp:835] [Rank 6] NCCL watchdog thread started! I1109 12:05:43.847890 30601 ProcessGroupNCCL.cpp:669] [Rank 6] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.847877 30600 ProcessGroupNCCL.cpp:669] [Rank 5] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.847914 30599 ProcessGroupNCCL.cpp:669] [Rank 4] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.847930 30602 ProcessGroupNCCL.cpp:669] [Rank 7] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.915059 31209 ProcessGroupNCCL.cpp:669] [Rank 62] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.915076 31206 ProcessGroupNCCL.cpp:669] [Rank 60] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.915088 31207 ProcessGroupNCCL.cpp:669] [Rank 61] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.915100 31210 ProcessGroupNCCL.cpp:669] [Rank 63] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.902462 23025 ProcessGroupNCCL.cpp:669] [Rank 46] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.902482 23024 ProcessGroupNCCL.cpp:669] [Rank 45] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.902496 23026 ProcessGroupNCCL.cpp:669] [Rank 47] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.902506 23023 ProcessGroupNCCL.cpp:669] [Rank 44] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.903659 16527 ProcessGroupNCCL.cpp:669] [Rank 53] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.903671 16526 ProcessGroupNCCL.cpp:669] [Rank 52] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.919378 6048 ProcessGroupNCCL.cpp:669] [Rank 94] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.903695 16528 ProcessGroupNCCL.cpp:669] [Rank 54] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.903695 16529 ProcessGroupNCCL.cpp:669] [Rank 55] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.919389 6046 ProcessGroupNCCL.cpp:669] [Rank 92] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.919406 6049 ProcessGroupNCCL.cpp:669] [Rank 95] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.919417 6047 ProcessGroupNCCL.cpp:669] [Rank 93] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.915283 16038 ProcessGroupNCCL.cpp:669] [Rank 58] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.915297 16037 ProcessGroupNCCL.cpp:669] [Rank 57] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.915309 16039 ProcessGroupNCCL.cpp:669] [Rank 59] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.915318 16035 ProcessGroupNCCL.cpp:669] [Rank 56] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.898797 22275 ProcessGroupNCCL.cpp:669] [Rank 64] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.898820 22277 ProcessGroupNCCL.cpp:669] [Rank 66] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.898936 22276 ProcessGroupNCCL.cpp:669] [Rank 65] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.899005 22278 ProcessGroupNCCL.cpp:669] [Rank 67] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.921502 29372 ProcessGroupNCCL.cpp:669] [Rank 50] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.921516 29373 ProcessGroupNCCL.cpp:669] [Rank 51] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.921533 29369 ProcessGroupNCCL.cpp:669] [Rank 48] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.918622 26608 ProcessGroupNCCL.cpp:669] [Rank 32] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.921542 29371 ProcessGroupNCCL.cpp:669] [Rank 49] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.918643 26609 ProcessGroupNCCL.cpp:669] [Rank 33] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.918658 26611 ProcessGroupNCCL.cpp:669] [Rank 35] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.918660 26610 ProcessGroupNCCL.cpp:669] [Rank 34] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.906291 4756 ProcessGroupNCCL.cpp:669] [Rank 16] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.924923 11608 ProcessGroupNCCL.cpp:669] [Rank 37] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.924940 11609 ProcessGroupNCCL.cpp:669] [Rank 38] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.924954 11607 ProcessGroupNCCL.cpp:669] [Rank 36] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.924971 11610 ProcessGroupNCCL.cpp:669] [Rank 39] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.924710 2109 ProcessGroupNCCL.cpp:669] [Rank 85] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.924746 2111 ProcessGroupNCCL.cpp:669] [Rank 87] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.924723 2107 ProcessGroupNCCL.cpp:669] [Rank 84] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.924754 2110 ProcessGroupNCCL.cpp:669] [Rank 86] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.909556 4760 ProcessGroupNCCL.cpp:669] [Rank 19] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.911442 4758 ProcessGroupNCCL.cpp:669] [Rank 17] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.913373 4759 ProcessGroupNCCL.cpp:669] [Rank 18] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:44.008632 25919 ProcessGroupNCCL.cpp:669] [Rank 82] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:44.008651 25917 ProcessGroupNCCL.cpp:669] [Rank 80] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:44.008661 25918 ProcessGroupNCCL.cpp:669] [Rank 81] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:44.008675 25920 ProcessGroupNCCL.cpp:669] [Rank 83] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.989769 629 ProcessGroupNCCL.cpp:669] [Rank 15] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.989904 627 ProcessGroupNCCL.cpp:669] [Rank 13] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.989931 628 ProcessGroupNCCL.cpp:669] [Rank 14] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.989943 626 ProcessGroupNCCL.cpp:669] [Rank 12] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.992367 15187 ProcessGroupNCCL.cpp:669] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.992380 15188 ProcessGroupNCCL.cpp:669] [Rank 2] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.992398 15185 ProcessGroupNCCL.cpp:669] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:05:43.992398 15189 ProcessGroupNCCL.cpp:669] [Rank 3] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. I1109 12:05:48.245900 15185 ProcessGroupNCCL.cpp:1274] NCCL_DEBUG: INFO Loading checkpoint shards: 0%| | 0/2 [00:00 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors I1109 12:07:08.279536 488 ProcessGroupNCCL.cpp:835] [Rank 42] NCCL watchdog thread started! I1109 12:07:08.279489 30379 ProcessGroupNCCL.cpp:669] [Rank 42] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.281185 21864 ProcessGroupNCCL.cpp:669] [Rank 21] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.281291 24365 ProcessGroupNCCL.cpp:835] [Rank 21] NCCL watchdog thread started! I1109 12:07:08.281317 24366 ProcessGroupNCCL.cpp:835] [Rank 20] NCCL watchdog thread started! I1109 12:07:08.281262 21863 ProcessGroupNCCL.cpp:669] [Rank 20] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.286374 1827 ProcessGroupNCCL.cpp:669] [Rank 25] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.286486 4426 ProcessGroupNCCL.cpp:835] [Rank 25] NCCL watchdog thread started! I1109 12:07:08.286490 4427 ProcessGroupNCCL.cpp:835] [Rank 24] NCCL watchdog thread started! I1109 12:07:08.286442 1826 ProcessGroupNCCL.cpp:669] [Rank 24] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.279827 489 ProcessGroupNCCL.cpp:835] [Rank 43] NCCL watchdog thread started! I1109 12:07:08.286686 4428 ProcessGroupNCCL.cpp:835] [Rank 26] NCCL watchdog thread started! I1109 12:07:08.279753 30380 ProcessGroupNCCL.cpp:669] [Rank 43] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.281432 21865 ProcessGroupNCCL.cpp:669] [Rank 22] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.286625 1828 ProcessGroupNCCL.cpp:669] [Rank 26] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.279861 30378 ProcessGroupNCCL.cpp:669] [Rank 41] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.281545 24367 ProcessGroupNCCL.cpp:835] [Rank 22] NCCL watchdog thread started! I1109 12:07:08.280004 490 ProcessGroupNCCL.cpp:835] [Rank 41] NCCL watchdog thread started! I1109 12:07:08.280022 492 ProcessGroupNCCL.cpp:835] [Rank 40] NCCL watchdog thread started! I1109 12:07:08.281631 21866 ProcessGroupNCCL.cpp:669] [Rank 23] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.279981 30377 ProcessGroupNCCL.cpp:669] [Rank 40] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.281689 24368 ProcessGroupNCCL.cpp:835] [Rank 23] NCCL watchdog thread started! I1109 12:07:08.286829 1829 ProcessGroupNCCL.cpp:669] [Rank 27] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.280354 28148 ProcessGroupNCCL.cpp:669] [Rank 70] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.280426 30691 ProcessGroupNCCL.cpp:835] [Rank 70] NCCL watchdog thread started! I1109 12:07:08.281636 31187 ProcessGroupNCCL.cpp:669] [Rank 79] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.281746 822 ProcessGroupNCCL.cpp:835] [Rank 79] NCCL watchdog thread started! I1109 12:07:08.286958 4429 ProcessGroupNCCL.cpp:835] [Rank 27] NCCL watchdog thread started! I1109 12:07:08.281715 31186 ProcessGroupNCCL.cpp:669] [Rank 78] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.281827 823 ProcessGroupNCCL.cpp:835] [Rank 78] NCCL watchdog thread started! I1109 12:07:08.281765 31185 ProcessGroupNCCL.cpp:669] [Rank 77] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.281867 824 ProcessGroupNCCL.cpp:835] [Rank 77] NCCL watchdog thread started! I1109 12:07:08.280580 28150 ProcessGroupNCCL.cpp:669] [Rank 71] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.280694 30692 ProcessGroupNCCL.cpp:835] [Rank 71] NCCL watchdog thread started! I1109 12:07:08.288532 4758 ProcessGroupNCCL.cpp:669] [Rank 17] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.288658 7214 ProcessGroupNCCL.cpp:835] [Rank 17] NCCL watchdog thread started! I1109 12:07:08.288678 7215 ProcessGroupNCCL.cpp:835] [Rank 16] NCCL watchdog thread started! I1109 12:07:08.288657 4756 ProcessGroupNCCL.cpp:669] [Rank 16] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.280786 30693 ProcessGroupNCCL.cpp:835] [Rank 68] NCCL watchdog thread started! I1109 12:07:08.282076 31184 ProcessGroupNCCL.cpp:669] [Rank 76] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.288738 4759 ProcessGroupNCCL.cpp:669] [Rank 18] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.288847 7216 ProcessGroupNCCL.cpp:835] [Rank 18] NCCL watchdog thread started! I1109 12:07:08.282176 825 ProcessGroupNCCL.cpp:835] [Rank 76] NCCL watchdog thread started! I1109 12:07:08.280437 11608 ProcessGroupNCCL.cpp:669] [Rank 37] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.280534 14296 ProcessGroupNCCL.cpp:835] [Rank 37] NCCL watchdog thread started! I1109 12:07:08.280761 28145 ProcessGroupNCCL.cpp:669] [Rank 68] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.288854 4760 ProcessGroupNCCL.cpp:669] [Rank 19] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.288935 7217 ProcessGroupNCCL.cpp:835] [Rank 19] NCCL watchdog thread started! I1109 12:07:08.288336 15188 ProcessGroupNCCL.cpp:669] [Rank 2] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.280875 28147 ProcessGroupNCCL.cpp:669] [Rank 69] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.280987 30694 ProcessGroupNCCL.cpp:835] [Rank 69] NCCL watchdog thread started! I1109 12:07:08.288458 15189 ProcessGroupNCCL.cpp:669] [Rank 3] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.288558 17945 ProcessGroupNCCL.cpp:835] [Rank 3] NCCL watchdog thread started! I1109 12:07:08.280670 11609 ProcessGroupNCCL.cpp:669] [Rank 38] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.288497 17944 ProcessGroupNCCL.cpp:835] [Rank 2] NCCL watchdog thread started! I1109 12:07:08.280702 11610 ProcessGroupNCCL.cpp:669] [Rank 39] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.288678 17946 ProcessGroupNCCL.cpp:835] [Rank 1] NCCL watchdog thread started! I1109 12:07:08.288620 15187 ProcessGroupNCCL.cpp:669] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.280814 14298 ProcessGroupNCCL.cpp:835] [Rank 39] NCCL watchdog thread started! I1109 12:07:08.280835 14297 ProcessGroupNCCL.cpp:835] [Rank 38] NCCL watchdog thread started! I1109 12:07:08.280948 11607 ProcessGroupNCCL.cpp:669] [Rank 36] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.281042 14299 ProcessGroupNCCL.cpp:835] [Rank 36] NCCL watchdog thread started! I1109 12:07:08.289314 15185 ProcessGroupNCCL.cpp:669] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.287496 26998 ProcessGroupNCCL.cpp:669] [Rank 75] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.287528 29506 ProcessGroupNCCL.cpp:835] [Rank 75] NCCL watchdog thread started! I1109 12:07:08.289446 17947 ProcessGroupNCCL.cpp:835] [Rank 0] NCCL watchdog thread started! I1109 12:07:08.287611 29507 ProcessGroupNCCL.cpp:835] [Rank 72] NCCL watchdog thread started! I1109 12:07:08.287566 26995 ProcessGroupNCCL.cpp:669] [Rank 72] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.287781 29508 ProcessGroupNCCL.cpp:835] [Rank 73] NCCL watchdog thread started! I1109 12:07:08.287703 26996 ProcessGroupNCCL.cpp:669] [Rank 73] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.287869 26997 ProcessGroupNCCL.cpp:669] [Rank 74] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.287973 29509 ProcessGroupNCCL.cpp:835] [Rank 74] NCCL watchdog thread started! I1109 12:07:08.287228 6049 ProcessGroupNCCL.cpp:669] [Rank 95] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.287282 6047 ProcessGroupNCCL.cpp:669] [Rank 93] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.287370 8972 ProcessGroupNCCL.cpp:835] [Rank 93] NCCL watchdog thread started! I1109 12:07:08.287458 8973 ProcessGroupNCCL.cpp:835] [Rank 92] NCCL watchdog thread started! I1109 12:07:08.287418 6046 ProcessGroupNCCL.cpp:669] [Rank 92] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.287357 8971 ProcessGroupNCCL.cpp:835] [Rank 95] NCCL watchdog thread started! I1109 12:07:08.287482 8974 ProcessGroupNCCL.cpp:835] [Rank 94] NCCL watchdog thread started! I1109 12:07:08.287478 6048 ProcessGroupNCCL.cpp:669] [Rank 94] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.294358 25919 ProcessGroupNCCL.cpp:669] [Rank 82] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.294483 28023 ProcessGroupNCCL.cpp:835] [Rank 82] NCCL watchdog thread started! I1109 12:07:08.294492 28024 ProcessGroupNCCL.cpp:835] [Rank 81] NCCL watchdog thread started! I1109 12:07:08.294421 25918 ProcessGroupNCCL.cpp:669] [Rank 81] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.294567 25920 ProcessGroupNCCL.cpp:669] [Rank 83] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.294677 28025 ProcessGroupNCCL.cpp:835] [Rank 83] NCCL watchdog thread started! I1109 12:07:08.294735 28026 ProcessGroupNCCL.cpp:835] [Rank 80] NCCL watchdog thread started! I1109 12:07:08.294695 25917 ProcessGroupNCCL.cpp:669] [Rank 80] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.338613 2109 ProcessGroupNCCL.cpp:669] [Rank 85] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.338709 4860 ProcessGroupNCCL.cpp:835] [Rank 85] NCCL watchdog thread started! I1109 12:07:08.338763 4861 ProcessGroupNCCL.cpp:835] [Rank 84] NCCL watchdog thread started! I1109 12:07:08.338722 2107 ProcessGroupNCCL.cpp:669] [Rank 84] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.338838 2111 ProcessGroupNCCL.cpp:669] [Rank 87] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.338961 4862 ProcessGroupNCCL.cpp:835] [Rank 87] NCCL watchdog thread started! I1109 12:07:08.338989 4863 ProcessGroupNCCL.cpp:835] [Rank 86] NCCL watchdog thread started! I1109 12:07:08.338980 2110 ProcessGroupNCCL.cpp:669] [Rank 86] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.354326 26610 ProcessGroupNCCL.cpp:669] [Rank 34] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.354342 26609 ProcessGroupNCCL.cpp:669] [Rank 33] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.354434 28676 ProcessGroupNCCL.cpp:835] [Rank 34] NCCL watchdog thread started! I1109 12:07:08.354470 28677 ProcessGroupNCCL.cpp:835] [Rank 33] NCCL watchdog thread started! I1109 12:07:08.354533 26611 ProcessGroupNCCL.cpp:669] [Rank 35] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.354655 28678 ProcessGroupNCCL.cpp:835] [Rank 35] NCCL watchdog thread started! I1109 12:07:08.354575 26608 ProcessGroupNCCL.cpp:669] [Rank 32] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.354676 28679 ProcessGroupNCCL.cpp:835] [Rank 32] NCCL watchdog thread started! I1109 12:07:08.358031 16037 ProcessGroupNCCL.cpp:669] [Rank 57] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.358090 16039 ProcessGroupNCCL.cpp:669] [Rank 59] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.358119 18120 ProcessGroupNCCL.cpp:835] [Rank 57] NCCL watchdog thread started! I1109 12:07:08.358218 18121 ProcessGroupNCCL.cpp:835] [Rank 59] NCCL watchdog thread started! I1109 12:07:08.358311 16038 ProcessGroupNCCL.cpp:669] [Rank 58] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.358459 18122 ProcessGroupNCCL.cpp:835] [Rank 58] NCCL watchdog thread started! I1109 12:07:08.358538 16035 ProcessGroupNCCL.cpp:669] [Rank 56] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.358603 18123 ProcessGroupNCCL.cpp:835] [Rank 56] NCCL watchdog thread started! I1109 12:07:08.364681 31209 ProcessGroupNCCL.cpp:669] [Rank 62] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.364761 914 ProcessGroupNCCL.cpp:835] [Rank 62] NCCL watchdog thread started! I1109 12:07:08.364889 3549 ProcessGroupNCCL.cpp:835] [Rank 28] NCCL watchdog thread started! I1109 12:07:08.364820 906 ProcessGroupNCCL.cpp:669] [Rank 28] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.364773 31210 ProcessGroupNCCL.cpp:669] [Rank 63] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.364806 907 ProcessGroupNCCL.cpp:669] [Rank 29] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.364924 3548 ProcessGroupNCCL.cpp:835] [Rank 29] NCCL watchdog thread started! I1109 12:07:08.364878 31207 ProcessGroupNCCL.cpp:669] [Rank 61] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.364895 915 ProcessGroupNCCL.cpp:835] [Rank 63] NCCL watchdog thread started! I1109 12:07:08.364919 908 ProcessGroupNCCL.cpp:669] [Rank 30] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.364993 916 ProcessGroupNCCL.cpp:835] [Rank 61] NCCL watchdog thread started! I1109 12:07:08.365032 3550 ProcessGroupNCCL.cpp:835] [Rank 30] NCCL watchdog thread started! I1109 12:07:08.365072 917 ProcessGroupNCCL.cpp:835] [Rank 60] NCCL watchdog thread started! I1109 12:07:08.364976 909 ProcessGroupNCCL.cpp:669] [Rank 31] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.365075 3551 ProcessGroupNCCL.cpp:835] [Rank 31] NCCL watchdog thread started! I1109 12:07:08.365000 31206 ProcessGroupNCCL.cpp:669] [Rank 60] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.369995 16528 ProcessGroupNCCL.cpp:669] [Rank 54] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.370108 19035 ProcessGroupNCCL.cpp:835] [Rank 54] NCCL watchdog thread started! I1109 12:07:08.370074 16527 ProcessGroupNCCL.cpp:669] [Rank 53] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.370175 19036 ProcessGroupNCCL.cpp:835] [Rank 53] NCCL watchdog thread started! I1109 12:07:08.370452 16526 ProcessGroupNCCL.cpp:669] [Rank 52] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.370487 19038 ProcessGroupNCCL.cpp:835] [Rank 52] NCCL watchdog thread started! I1109 12:07:08.370424 16529 ProcessGroupNCCL.cpp:669] [Rank 55] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.370527 19037 ProcessGroupNCCL.cpp:835] [Rank 55] NCCL watchdog thread started! I1109 12:07:08.371943 22276 ProcessGroupNCCL.cpp:669] [Rank 65] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.372046 24734 ProcessGroupNCCL.cpp:835] [Rank 65] NCCL watchdog thread started! I1109 12:07:08.372171 24735 ProcessGroupNCCL.cpp:835] [Rank 64] NCCL watchdog thread started! I1109 12:07:08.372108 22275 ProcessGroupNCCL.cpp:669] [Rank 64] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.372331 22278 ProcessGroupNCCL.cpp:669] [Rank 67] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.372453 24736 ProcessGroupNCCL.cpp:835] [Rank 67] NCCL watchdog thread started! I1109 12:07:08.372433 22277 ProcessGroupNCCL.cpp:669] [Rank 66] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.372527 24737 ProcessGroupNCCL.cpp:835] [Rank 66] NCCL watchdog thread started! I1109 12:07:08.368604 627 ProcessGroupNCCL.cpp:669] [Rank 13] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.368707 3581 ProcessGroupNCCL.cpp:835] [Rank 13] NCCL watchdog thread started! I1109 12:07:08.368700 3582 ProcessGroupNCCL.cpp:835] [Rank 12] NCCL watchdog thread started! I1109 12:07:08.368662 626 ProcessGroupNCCL.cpp:669] [Rank 12] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.368832 628 ProcessGroupNCCL.cpp:669] [Rank 14] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.368942 3583 ProcessGroupNCCL.cpp:835] [Rank 14] NCCL watchdog thread started! I1109 12:07:08.368960 3584 ProcessGroupNCCL.cpp:835] [Rank 15] NCCL watchdog thread started! I1109 12:07:08.368876 629 ProcessGroupNCCL.cpp:669] [Rank 15] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.374657 30600 ProcessGroupNCCL.cpp:669] [Rank 5] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.374732 300 ProcessGroupNCCL.cpp:835] [Rank 5] NCCL watchdog thread started! I1109 12:07:08.374866 30601 ProcessGroupNCCL.cpp:669] [Rank 6] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.375032 301 ProcessGroupNCCL.cpp:835] [Rank 6] NCCL watchdog thread started! I1109 12:07:08.375046 302 ProcessGroupNCCL.cpp:835] [Rank 4] NCCL watchdog thread started! I1109 12:07:08.374991 30599 ProcessGroupNCCL.cpp:669] [Rank 4] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.375012 30602 ProcessGroupNCCL.cpp:669] [Rank 7] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.375111 303 ProcessGroupNCCL.cpp:835] [Rank 7] NCCL watchdog thread started! I1109 12:07:08.379796 24492 ProcessGroupNCCL.cpp:669] [Rank 90] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.379880 27105 ProcessGroupNCCL.cpp:835] [Rank 90] NCCL watchdog thread started! I1109 12:07:08.373006 25692 ProcessGroupNCCL.cpp:835] [Rank 8] NCCL watchdog thread started! I1109 12:07:08.372959 23099 ProcessGroupNCCL.cpp:669] [Rank 8] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.380694 25564 ProcessGroupNCCL.cpp:835] [Rank 45] NCCL watchdog thread started! I1109 12:07:08.380645 23024 ProcessGroupNCCL.cpp:669] [Rank 45] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.379999 24491 ProcessGroupNCCL.cpp:669] [Rank 89] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.379987 24490 ProcessGroupNCCL.cpp:669] [Rank 88] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.380123 27107 ProcessGroupNCCL.cpp:835] [Rank 89] NCCL watchdog thread started! I1109 12:07:08.380102 27106 ProcessGroupNCCL.cpp:835] [Rank 88] NCCL watchdog thread started! I1109 12:07:08.373227 23101 ProcessGroupNCCL.cpp:669] [Rank 9] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.373335 25693 ProcessGroupNCCL.cpp:835] [Rank 9] NCCL watchdog thread started! I1109 12:07:08.380844 23023 ProcessGroupNCCL.cpp:669] [Rank 44] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.380939 25565 ProcessGroupNCCL.cpp:835] [Rank 44] NCCL watchdog thread started! I1109 12:07:08.380192 24493 ProcessGroupNCCL.cpp:669] [Rank 91] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.380986 25566 ProcessGroupNCCL.cpp:835] [Rank 46] NCCL watchdog thread started! I1109 12:07:08.380287 27108 ProcessGroupNCCL.cpp:835] [Rank 91] NCCL watchdog thread started! I1109 12:07:08.380942 23026 ProcessGroupNCCL.cpp:669] [Rank 47] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.373502 25694 ProcessGroupNCCL.cpp:835] [Rank 11] NCCL watchdog thread started! I1109 12:07:08.380899 23025 ProcessGroupNCCL.cpp:669] [Rank 46] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.373447 23103 ProcessGroupNCCL.cpp:669] [Rank 11] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.381052 25567 ProcessGroupNCCL.cpp:835] [Rank 47] NCCL watchdog thread started! I1109 12:07:08.373584 23102 ProcessGroupNCCL.cpp:669] [Rank 10] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.373701 25695 ProcessGroupNCCL.cpp:835] [Rank 10] NCCL watchdog thread started! I1109 12:07:08.383754 29371 ProcessGroupNCCL.cpp:669] [Rank 49] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.383862 31528 ProcessGroupNCCL.cpp:835] [Rank 49] NCCL watchdog thread started! I1109 12:07:08.384107 29373 ProcessGroupNCCL.cpp:669] [Rank 51] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.384075 29372 ProcessGroupNCCL.cpp:669] [Rank 50] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1109 12:07:08.384225 31529 ProcessGroupNCCL.cpp:835] [Rank 50] NCCL watchdog thread started! I1109 12:07:08.384222 31530 ProcessGroupNCCL.cpp:835] [Rank 51] NCCL watchdog thread started! I1109 12:07:08.384248 31531 ProcessGroupNCCL.cpp:835] [Rank 48] NCCL watchdog thread started! I1109 12:07:08.384155 29369 ProcessGroupNCCL.cpp:669] [Rank 48] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 0%| | 0/420 [00:00