WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.200500 25528 ProcessGroupNCCL.cpp:835] [Rank 69] NCCL watchdog thread started! I1027 12:39:33.200505 24733 ProcessGroupNCCL.cpp:669] [Rank 69] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.210026 27671 ProcessGroupNCCL.cpp:835] [Rank 5] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.200668 25529 ProcessGroupNCCL.cpp:835] [Rank 68] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.210045 27673 ProcessGroupNCCL.cpp:835] [Rank 4] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.209512 30936 ProcessGroupNCCL.cpp:835] [Rank 56] NCCL watchdog thread started! I1027 12:39:33.200650 24731 ProcessGroupNCCL.cpp:669] [Rank 68] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.210021 27004 ProcessGroupNCCL.cpp:669] [Rank 5] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.209606 30937 ProcessGroupNCCL.cpp:835] [Rank 57] NCCL watchdog thread started! I1027 12:39:33.210036 27002 ProcessGroupNCCL.cpp:669] [Rank 4] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.210299 30939 ProcessGroupNCCL.cpp:835] [Rank 59] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.211239 27675 ProcessGroupNCCL.cpp:835] [Rank 7] NCCL watchdog thread started! I1027 12:39:33.211216 27006 ProcessGroupNCCL.cpp:669] [Rank 7] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.201906 25531 ProcessGroupNCCL.cpp:835] [Rank 71] NCCL watchdog thread started! I1027 12:39:33.201882 24735 ProcessGroupNCCL.cpp:669] [Rank 71] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.212044 30401 ProcessGroupNCCL.cpp:669] [Rank 56] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.212057 30405 ProcessGroupNCCL.cpp:669] [Rank 59] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.212060 30403 ProcessGroupNCCL.cpp:669] [Rank 57] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.214677 18869 ProcessGroupNCCL.cpp:835] [Rank 25] NCCL watchdog thread started! I1027 12:39:33.214673 18194 ProcessGroupNCCL.cpp:669] [Rank 25] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.214730 18870 ProcessGroupNCCL.cpp:835] [Rank 24] NCCL watchdog thread started! I1027 12:39:33.214725 18192 ProcessGroupNCCL.cpp:669] [Rank 24] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.222362 18873 ProcessGroupNCCL.cpp:835] [Rank 27] NCCL watchdog thread started! I1027 12:39:33.222359 18196 ProcessGroupNCCL.cpp:669] [Rank 27] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.222620 18874 ProcessGroupNCCL.cpp:835] [Rank 26] NCCL watchdog thread started! I1027 12:39:33.222625 18195 ProcessGroupNCCL.cpp:669] [Rank 26] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.242220 9870 ProcessGroupNCCL.cpp:835] [Rank 45] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.242251 9871 ProcessGroupNCCL.cpp:835] [Rank 47] NCCL watchdog thread started! I1027 12:39:33.242194 9032 ProcessGroupNCCL.cpp:669] [Rank 45] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.242238 9034 ProcessGroupNCCL.cpp:669] [Rank 47] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.242344 9872 ProcessGroupNCCL.cpp:835] [Rank 44] NCCL watchdog thread started! I1027 12:39:33.242318 9030 ProcessGroupNCCL.cpp:669] [Rank 44] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.232339 9430 ProcessGroupNCCL.cpp:835] [Rank 87] NCCL watchdog thread started! I1027 12:39:33.232316 8679 ProcessGroupNCCL.cpp:669] [Rank 87] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.236939 1589 ProcessGroupNCCL.cpp:835] [Rank 43] NCCL watchdog thread started! I1027 12:39:33.236860 755 ProcessGroupNCCL.cpp:669] [Rank 43] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.237089 1590 ProcessGroupNCCL.cpp:835] [Rank 41] NCCL watchdog thread started! I1027 12:39:33.237085 753 ProcessGroupNCCL.cpp:669] [Rank 41] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.237143 1591 ProcessGroupNCCL.cpp:835] [Rank 40] NCCL watchdog thread started! I1027 12:39:33.237133 751 ProcessGroupNCCL.cpp:669] [Rank 40] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.253125 23552 ProcessGroupNCCL.cpp:835] [Rank 61] NCCL watchdog thread started! I1027 12:39:33.253119 22737 ProcessGroupNCCL.cpp:669] [Rank 61] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.253175 23553 ProcessGroupNCCL.cpp:835] [Rank 60] NCCL watchdog thread started! I1027 12:39:33.253170 22735 ProcessGroupNCCL.cpp:669] [Rank 60] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.266911 14285 ProcessGroupNCCL.cpp:835] [Rank 81] NCCL watchdog thread started! I1027 12:39:33.266911 13436 ProcessGroupNCCL.cpp:669] [Rank 81] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.267041 14286 ProcessGroupNCCL.cpp:835] [Rank 80] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.252874 19537 ProcessGroupNCCL.cpp:835] [Rank 91] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.256489 30456 ProcessGroupNCCL.cpp:835] [Rank 95] NCCL watchdog thread started! I1027 12:39:33.267035 13434 ProcessGroupNCCL.cpp:669] [Rank 80] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.252851 18788 ProcessGroupNCCL.cpp:669] [Rank 91] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.256574 30457 ProcessGroupNCCL.cpp:835] [Rank 92] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.256603 30455 ProcessGroupNCCL.cpp:835] [Rank 93] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.263026 6243 ProcessGroupNCCL.cpp:835] [Rank 3] NCCL watchdog thread started! I1027 12:39:33.263000 5142 ProcessGroupNCCL.cpp:669] [Rank 3] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.254642 1596 ProcessGroupNCCL.cpp:835] [Rank 8] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.266870 24886 ProcessGroupNCCL.cpp:835] [Rank 21] NCCL watchdog thread started! I1027 12:39:33.254657 872 ProcessGroupNCCL.cpp:669] [Rank 8] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.266855 23993 ProcessGroupNCCL.cpp:669] [Rank 21] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.254819 1597 ProcessGroupNCCL.cpp:835] [Rank 11] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.266937 24887 ProcessGroupNCCL.cpp:835] [Rank 20] NCCL watchdog thread started! I1027 12:39:33.266932 23991 ProcessGroupNCCL.cpp:669] [Rank 20] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.254792 876 ProcessGroupNCCL.cpp:669] [Rank 11] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.265645 6478 ProcessGroupNCCL.cpp:835] [Rank 15] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.266599 15174 ProcessGroupNCCL.cpp:835] [Rank 48] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.254907 1598 ProcessGroupNCCL.cpp:835] [Rank 9] NCCL watchdog thread started! I1027 12:39:33.254902 874 ProcessGroupNCCL.cpp:669] [Rank 9] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.265635 5734 ProcessGroupNCCL.cpp:669] [Rank 15] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.266575 14532 ProcessGroupNCCL.cpp:669] [Rank 48] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.266705 14536 ProcessGroupNCCL.cpp:669] [Rank 51] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.266729 15176 ProcessGroupNCCL.cpp:835] [Rank 51] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.266742 15177 ProcessGroupNCCL.cpp:835] [Rank 49] NCCL watchdog thread started! I1027 12:39:33.266741 14534 ProcessGroupNCCL.cpp:669] [Rank 49] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.258224 29649 ProcessGroupNCCL.cpp:669] [Rank 95] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.258237 29645 ProcessGroupNCCL.cpp:669] [Rank 92] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.258253 29647 ProcessGroupNCCL.cpp:669] [Rank 93] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.399395 23556 ProcessGroupNCCL.cpp:835] [Rank 63] NCCL watchdog thread started! I1027 12:39:33.399389 22739 ProcessGroupNCCL.cpp:669] [Rank 63] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.402199 30459 ProcessGroupNCCL.cpp:835] [Rank 94] NCCL watchdog thread started! I1027 12:39:33.402184 29648 ProcessGroupNCCL.cpp:669] [Rank 94] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.400110 1600 ProcessGroupNCCL.cpp:835] [Rank 10] NCCL watchdog thread started! I1027 12:39:33.400103 875 ProcessGroupNCCL.cpp:669] [Rank 10] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.402047 1593 ProcessGroupNCCL.cpp:835] [Rank 42] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.398943 19540 ProcessGroupNCCL.cpp:835] [Rank 90] NCCL watchdog thread started! I1027 12:39:33.402038 754 ProcessGroupNCCL.cpp:669] [Rank 42] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.398936 18787 ProcessGroupNCCL.cpp:669] [Rank 90] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.402812 9433 ProcessGroupNCCL.cpp:835] [Rank 86] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.412076 15179 ProcessGroupNCCL.cpp:835] [Rank 50] NCCL watchdog thread started! I1027 12:39:33.402804 8678 ProcessGroupNCCL.cpp:669] [Rank 86] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.412067 14535 ProcessGroupNCCL.cpp:669] [Rank 50] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.408944 6245 ProcessGroupNCCL.cpp:835] [Rank 2] NCCL watchdog thread started! I1027 12:39:33.408933 5141 ProcessGroupNCCL.cpp:669] [Rank 2] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.411923 27677 ProcessGroupNCCL.cpp:835] [Rank 6] NCCL watchdog thread started! I1027 12:39:33.411912 27005 ProcessGroupNCCL.cpp:669] [Rank 6] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.411231 6481 ProcessGroupNCCL.cpp:835] [Rank 14] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.413463 14289 ProcessGroupNCCL.cpp:835] [Rank 83] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.411335 30942 ProcessGroupNCCL.cpp:835] [Rank 58] NCCL watchdog thread started! I1027 12:39:33.411332 30404 ProcessGroupNCCL.cpp:669] [Rank 58] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.412562 24889 ProcessGroupNCCL.cpp:835] [Rank 22] NCCL watchdog thread started! I1027 12:39:33.411221 5733 ProcessGroupNCCL.cpp:669] [Rank 14] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.413463 13438 ProcessGroupNCCL.cpp:669] [Rank 83] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.412555 23994 ProcessGroupNCCL.cpp:669] [Rank 22] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.413718 14290 ProcessGroupNCCL.cpp:835] [Rank 82] NCCL watchdog thread started! I1027 12:39:33.413703 13437 ProcessGroupNCCL.cpp:669] [Rank 82] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.413848 24891 ProcessGroupNCCL.cpp:835] [Rank 23] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.414569 9874 ProcessGroupNCCL.cpp:835] [Rank 46] NCCL watchdog thread started! I1027 12:39:33.413846 23995 ProcessGroupNCCL.cpp:669] [Rank 23] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.414562 9033 ProcessGroupNCCL.cpp:669] [Rank 46] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.401870 23558 ProcessGroupNCCL.cpp:835] [Rank 62] NCCL watchdog thread started! I1027 12:39:33.401856 22738 ProcessGroupNCCL.cpp:669] [Rank 62] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.404008 25534 ProcessGroupNCCL.cpp:835] [Rank 70] NCCL watchdog thread started! I1027 12:39:33.403996 24734 ProcessGroupNCCL.cpp:669] [Rank 70] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.410174 14309 ProcessGroupNCCL.cpp:835] [Rank 35] NCCL watchdog thread started! I1027 12:39:33.410156 13446 ProcessGroupNCCL.cpp:669] [Rank 35] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.406589 22700 ProcessGroupNCCL.cpp:835] [Rank 31] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.406713 22701 ProcessGroupNCCL.cpp:835] [Rank 28] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.410322 14310 ProcessGroupNCCL.cpp:835] [Rank 34] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.410351 14311 ProcessGroupNCCL.cpp:835] [Rank 32] NCCL watchdog thread started! I1027 12:39:33.410310 13445 ProcessGroupNCCL.cpp:669] [Rank 34] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.406581 22025 ProcessGroupNCCL.cpp:669] [Rank 31] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.410336 13442 ProcessGroupNCCL.cpp:669] [Rank 32] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.410470 14312 ProcessGroupNCCL.cpp:835] [Rank 33] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.406800 22023 ProcessGroupNCCL.cpp:669] [Rank 29] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.406879 22702 ProcessGroupNCCL.cpp:835] [Rank 29] NCCL watchdog thread started! I1027 12:39:33.410454 13444 ProcessGroupNCCL.cpp:669] [Rank 33] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.406702 22021 ProcessGroupNCCL.cpp:669] [Rank 28] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.407063 22703 ProcessGroupNCCL.cpp:835] [Rank 30] NCCL watchdog thread started! I1027 12:39:33.407068 22024 ProcessGroupNCCL.cpp:669] [Rank 30] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.405041 22145 ProcessGroupNCCL.cpp:835] [Rank 37] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.405190 22146 ProcessGroupNCCL.cpp:835] [Rank 36] NCCL watchdog thread started! I1027 12:39:33.405036 21282 ProcessGroupNCCL.cpp:669] [Rank 37] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.405184 21280 ProcessGroupNCCL.cpp:669] [Rank 36] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.405328 22147 ProcessGroupNCCL.cpp:835] [Rank 39] NCCL watchdog thread started! I1027 12:39:33.405305 21284 ProcessGroupNCCL.cpp:669] [Rank 39] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.405390 22148 ProcessGroupNCCL.cpp:835] [Rank 38] NCCL watchdog thread started! I1027 12:39:33.405385 21283 ProcessGroupNCCL.cpp:669] [Rank 38] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.414642 23013 ProcessGroupNCCL.cpp:835] [Rank 73] NCCL watchdog thread started! I1027 12:39:33.414642 22283 ProcessGroupNCCL.cpp:669] [Rank 73] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.414917 23014 ProcessGroupNCCL.cpp:835] [Rank 74] NCCL watchdog thread started! I1027 12:39:33.414911 22284 ProcessGroupNCCL.cpp:669] [Rank 74] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.414996 23015 ProcessGroupNCCL.cpp:835] [Rank 75] NCCL watchdog thread started! I1027 12:39:33.414973 22285 ProcessGroupNCCL.cpp:669] [Rank 75] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.415024 23016 ProcessGroupNCCL.cpp:835] [Rank 72] NCCL watchdog thread started! I1027 12:39:33.415022 22281 ProcessGroupNCCL.cpp:669] [Rank 72] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.419014 17475 ProcessGroupNCCL.cpp:835] [Rank 64] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.408730 5661 ProcessGroupNCCL.cpp:835] [Rank 76] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.419026 17476 ProcessGroupNCCL.cpp:835] [Rank 66] NCCL watchdog thread started! I1027 12:39:33.418990 16936 ProcessGroupNCCL.cpp:669] [Rank 64] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.408726 4889 ProcessGroupNCCL.cpp:669] [Rank 76] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.408809 5662 ProcessGroupNCCL.cpp:835] [Rank 77] NCCL watchdog thread started! I1027 12:39:33.408785 4891 ProcessGroupNCCL.cpp:669] [Rank 77] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.419132 17477 ProcessGroupNCCL.cpp:835] [Rank 67] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.419189 17478 ProcessGroupNCCL.cpp:835] [Rank 65] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.419503 9358 ProcessGroupNCCL.cpp:835] [Rank 19] NCCL watchdog thread started! I1027 12:39:33.419019 16939 ProcessGroupNCCL.cpp:669] [Rank 66] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.419644 9359 ProcessGroupNCCL.cpp:835] [Rank 17] NCCL watchdog thread started! I1027 12:39:33.419122 16940 ProcessGroupNCCL.cpp:669] [Rank 67] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.419441 8675 ProcessGroupNCCL.cpp:669] [Rank 19] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.419780 6972 ProcessGroupNCCL.cpp:835] [Rank 52] NCCL watchdog thread started! I1027 12:39:33.419167 16938 ProcessGroupNCCL.cpp:669] [Rank 65] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.419575 8673 ProcessGroupNCCL.cpp:669] [Rank 17] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.419750 6275 ProcessGroupNCCL.cpp:669] [Rank 52] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.419852 9360 ProcessGroupNCCL.cpp:835] [Rank 16] NCCL watchdog thread started! WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.419932 6973 ProcessGroupNCCL.cpp:835] [Rank 53] NCCL watchdog thread started! I1027 12:39:33.419845 8671 ProcessGroupNCCL.cpp:669] [Rank 16] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.419934 6974 ProcessGroupNCCL.cpp:835] [Rank 54] NCCL watchdog thread started! I1027 12:39:33.419927 6278 ProcessGroupNCCL.cpp:669] [Rank 54] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.419895 9361 ProcessGroupNCCL.cpp:835] [Rank 18] NCCL watchdog thread started! I1027 12:39:33.419906 6277 ProcessGroupNCCL.cpp:669] [Rank 53] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:39:33.419888 8674 ProcessGroupNCCL.cpp:669] [Rank 18] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:33.420027 6975 ProcessGroupNCCL.cpp:835] [Rank 55] NCCL watchdog thread started! I1027 12:39:33.420002 6279 ProcessGroupNCCL.cpp:669] [Rank 55] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:34.206053 6248 ProcessGroupNCCL.cpp:835] [Rank 1] NCCL watchdog thread started! I1027 12:39:34.206019 5140 ProcessGroupNCCL.cpp:669] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:34.200440 9446 ProcessGroupNCCL.cpp:835] [Rank 85] NCCL watchdog thread started! I1027 12:39:34.200433 8677 ProcessGroupNCCL.cpp:669] [Rank 85] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:34.208842 6485 ProcessGroupNCCL.cpp:835] [Rank 13] NCCL watchdog thread started! I1027 12:39:34.208811 5732 ProcessGroupNCCL.cpp:669] [Rank 13] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:34.208937 6486 ProcessGroupNCCL.cpp:835] [Rank 12] NCCL watchdog thread started! I1027 12:39:34.208930 5730 ProcessGroupNCCL.cpp:669] [Rank 12] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:34.196820 19558 ProcessGroupNCCL.cpp:835] [Rank 88] NCCL watchdog thread started! I1027 12:39:34.196811 18784 ProcessGroupNCCL.cpp:669] [Rank 88] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:34.196872 19559 ProcessGroupNCCL.cpp:835] [Rank 89] NCCL watchdog thread started! I1027 12:39:34.196848 18786 ProcessGroupNCCL.cpp:669] [Rank 89] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:34.200856 9447 ProcessGroupNCCL.cpp:835] [Rank 84] NCCL watchdog thread started! I1027 12:39:34.200851 8675 ProcessGroupNCCL.cpp:669] [Rank 84] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:34.201802 5681 ProcessGroupNCCL.cpp:835] [Rank 79] NCCL watchdog thread started! I1027 12:39:34.201779 4893 ProcessGroupNCCL.cpp:669] [Rank 79] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:34.201927 5682 ProcessGroupNCCL.cpp:835] [Rank 78] NCCL watchdog thread started! I1027 12:39:34.201921 4892 ProcessGroupNCCL.cpp:669] [Rank 78] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 WARNING: Logging before InitGoogleLogging() is written to STDERR I1027 12:39:34.215355 6250 ProcessGroupNCCL.cpp:835] [Rank 0] NCCL watchdog thread started! I1027 12:39:34.215348 5136 ProcessGroupNCCL.cpp:669] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision. pthread_mutex_timedlock() returned 110 Failed to initialize RSMI device mutex after 5 seconds. Previous execution may not have shutdown cleanly. To fix problem, stop all rocm_smi programs, and then delete the rocm_smi* shared memory files in /dev/shm.: Success pthread_mutex_timedlock() returned 110 Failed to initialize RSMI device mutex after 5 seconds. Previous execution may not have shutdown cleanly. To fix problem, stop all rocm_smi programs, and then delete the rocm_smi* shared memory files in /dev/shm.: Success pthread_mutex_timedlock() returned 110 Failed to initialize RSMI device mutex after 5 seconds. Previous execution may not have shutdown cleanly. To fix problem, stop all rocm_smi programs, and then delete the rocm_smi* shared memory files in /dev/shm.: Success pthread_mutex_timedlock() returned 110 Failed to initialize RSMI device mutex after 5 seconds. Previous execution may not have shutdown cleanly. To fix problem, stop all rocm_smi programs, and then delete the rocm_smi* shared memory files in /dev/shm.: Success rsmi_init() failed rsmi_init() failed rsmi_init() failed rsmi_init() failed I1027 12:39:42.055336 5136 ProcessGroupNCCL.cpp:1274] NCCL_DEBUG: INFO Loading checkpoint shards: 0%| | 0/2 [00:00 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (159 > 64). Running this sequence through the model will result in indexing errors I1027 12:40:47.601089 8679 ProcessGroupNCCL.cpp:669] [Rank 87] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.601212 10814 ProcessGroupNCCL.cpp:835] [Rank 87] NCCL watchdog thread started! I1027 12:40:47.601305 8675 ProcessGroupNCCL.cpp:669] [Rank 84] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.601416 10815 ProcessGroupNCCL.cpp:835] [Rank 84] NCCL watchdog thread started! I1027 12:40:47.601462 10816 ProcessGroupNCCL.cpp:835] [Rank 85] NCCL watchdog thread started! I1027 12:40:47.601398 8677 ProcessGroupNCCL.cpp:669] [Rank 85] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.601579 8678 ProcessGroupNCCL.cpp:669] [Rank 86] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.601651 10817 ProcessGroupNCCL.cpp:835] [Rank 86] NCCL watchdog thread started! I1027 12:40:47.599725 30405 ProcessGroupNCCL.cpp:669] [Rank 59] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.599853 32007 ProcessGroupNCCL.cpp:835] [Rank 59] NCCL watchdog thread started! I1027 12:40:47.601841 5733 ProcessGroupNCCL.cpp:669] [Rank 14] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.601965 7707 ProcessGroupNCCL.cpp:835] [Rank 14] NCCL watchdog thread started! I1027 12:40:47.600759 14534 ProcessGroupNCCL.cpp:669] [Rank 49] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.600880 16286 ProcessGroupNCCL.cpp:835] [Rank 49] NCCL watchdog thread started! I1027 12:40:47.600955 14536 ProcessGroupNCCL.cpp:669] [Rank 51] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.601065 16287 ProcessGroupNCCL.cpp:835] [Rank 51] NCCL watchdog thread started! I1027 12:40:47.599905 30404 ProcessGroupNCCL.cpp:669] [Rank 58] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.599989 32008 ProcessGroupNCCL.cpp:835] [Rank 58] NCCL watchdog thread started! I1027 12:40:47.601177 14535 ProcessGroupNCCL.cpp:669] [Rank 50] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.600229 32010 ProcessGroupNCCL.cpp:835] [Rank 57] NCCL watchdog thread started! I1027 12:40:47.600174 30403 ProcessGroupNCCL.cpp:669] [Rank 57] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.601302 16288 ProcessGroupNCCL.cpp:835] [Rank 50] NCCL watchdog thread started! I1027 12:40:47.600133 30401 ProcessGroupNCCL.cpp:669] [Rank 56] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.600250 32009 ProcessGroupNCCL.cpp:835] [Rank 56] NCCL watchdog thread started! I1027 12:40:47.602342 5730 ProcessGroupNCCL.cpp:669] [Rank 12] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.601234 14532 ProcessGroupNCCL.cpp:669] [Rank 48] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.601363 16289 ProcessGroupNCCL.cpp:835] [Rank 48] NCCL watchdog thread started! I1027 12:40:47.602416 5734 ProcessGroupNCCL.cpp:669] [Rank 15] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.602447 7708 ProcessGroupNCCL.cpp:835] [Rank 12] NCCL watchdog thread started! I1027 12:40:47.602535 7709 ProcessGroupNCCL.cpp:835] [Rank 15] NCCL watchdog thread started! I1027 12:40:47.602576 5732 ProcessGroupNCCL.cpp:669] [Rank 13] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.602665 7710 ProcessGroupNCCL.cpp:835] [Rank 13] NCCL watchdog thread started! I1027 12:40:47.601351 18192 ProcessGroupNCCL.cpp:669] [Rank 24] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.601472 20242 ProcessGroupNCCL.cpp:835] [Rank 24] NCCL watchdog thread started! I1027 12:40:47.601622 20243 ProcessGroupNCCL.cpp:835] [Rank 27] NCCL watchdog thread started! I1027 12:40:47.601575 18196 ProcessGroupNCCL.cpp:669] [Rank 27] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.601797 20244 ProcessGroupNCCL.cpp:835] [Rank 26] NCCL watchdog thread started! I1027 12:40:47.601732 18195 ProcessGroupNCCL.cpp:669] [Rank 26] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.592037 24735 ProcessGroupNCCL.cpp:669] [Rank 71] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.592154 26873 ProcessGroupNCCL.cpp:835] [Rank 71] NCCL watchdog thread started! I1027 12:40:47.592074 26872 ProcessGroupNCCL.cpp:835] [Rank 68] NCCL watchdog thread started! I1027 12:40:47.601945 20245 ProcessGroupNCCL.cpp:835] [Rank 25] NCCL watchdog thread started! I1027 12:40:47.601892 18194 ProcessGroupNCCL.cpp:669] [Rank 25] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.592162 26874 ProcessGroupNCCL.cpp:835] [Rank 70] NCCL watchdog thread started! I1027 12:40:47.592049 24734 ProcessGroupNCCL.cpp:669] [Rank 70] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.592013 24731 ProcessGroupNCCL.cpp:669] [Rank 68] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.592391 26875 ProcessGroupNCCL.cpp:835] [Rank 69] NCCL watchdog thread started! I1027 12:40:47.592339 24733 ProcessGroupNCCL.cpp:669] [Rank 69] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.605594 4893 ProcessGroupNCCL.cpp:669] [Rank 79] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.605691 6768 ProcessGroupNCCL.cpp:835] [Rank 79] NCCL watchdog thread started! I1027 12:40:47.605654 4892 ProcessGroupNCCL.cpp:669] [Rank 78] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.605765 6769 ProcessGroupNCCL.cpp:835] [Rank 78] NCCL watchdog thread started! I1027 12:40:47.605779 4891 ProcessGroupNCCL.cpp:669] [Rank 77] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.605890 6770 ProcessGroupNCCL.cpp:835] [Rank 77] NCCL watchdog thread started! I1027 12:40:47.604334 13446 ProcessGroupNCCL.cpp:669] [Rank 35] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.604588 15607 ProcessGroupNCCL.cpp:835] [Rank 34] NCCL watchdog thread started! I1027 12:40:47.604455 15606 ProcessGroupNCCL.cpp:835] [Rank 35] NCCL watchdog thread started! I1027 12:40:47.604584 13444 ProcessGroupNCCL.cpp:669] [Rank 33] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.606179 4889 ProcessGroupNCCL.cpp:669] [Rank 76] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.604686 15608 ProcessGroupNCCL.cpp:835] [Rank 33] NCCL watchdog thread started! I1027 12:40:47.604701 13442 ProcessGroupNCCL.cpp:669] [Rank 32] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.604790 15609 ProcessGroupNCCL.cpp:835] [Rank 32] NCCL watchdog thread started! I1027 12:40:47.604907 29647 ProcessGroupNCCL.cpp:669] [Rank 93] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.605041 31837 ProcessGroupNCCL.cpp:835] [Rank 93] NCCL watchdog thread started! I1027 12:40:47.606243 6771 ProcessGroupNCCL.cpp:835] [Rank 76] NCCL watchdog thread started! I1027 12:40:47.604502 13445 ProcessGroupNCCL.cpp:669] [Rank 34] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.605080 29649 ProcessGroupNCCL.cpp:669] [Rank 95] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.605175 31838 ProcessGroupNCCL.cpp:835] [Rank 95] NCCL watchdog thread started! I1027 12:40:47.603674 9030 ProcessGroupNCCL.cpp:669] [Rank 44] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.603740 11170 ProcessGroupNCCL.cpp:835] [Rank 44] NCCL watchdog thread started! I1027 12:40:47.605244 29648 ProcessGroupNCCL.cpp:669] [Rank 94] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.603852 9034 ProcessGroupNCCL.cpp:669] [Rank 47] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.605365 31839 ProcessGroupNCCL.cpp:835] [Rank 94] NCCL watchdog thread started! I1027 12:40:47.603915 11171 ProcessGroupNCCL.cpp:835] [Rank 47] NCCL watchdog thread started! I1027 12:40:47.605338 29645 ProcessGroupNCCL.cpp:669] [Rank 92] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.605454 31840 ProcessGroupNCCL.cpp:835] [Rank 92] NCCL watchdog thread started! I1027 12:40:47.604123 9032 ProcessGroupNCCL.cpp:669] [Rank 45] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.604238 11172 ProcessGroupNCCL.cpp:835] [Rank 45] NCCL watchdog thread started! I1027 12:40:47.604380 11173 ProcessGroupNCCL.cpp:835] [Rank 46] NCCL watchdog thread started! I1027 12:40:47.604375 9033 ProcessGroupNCCL.cpp:669] [Rank 46] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.607038 22024 ProcessGroupNCCL.cpp:669] [Rank 30] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.607158 23978 ProcessGroupNCCL.cpp:835] [Rank 30] NCCL watchdog thread started! I1027 12:40:47.603832 6279 ProcessGroupNCCL.cpp:669] [Rank 55] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.603924 7979 ProcessGroupNCCL.cpp:835] [Rank 55] NCCL watchdog thread started! I1027 12:40:47.607259 22021 ProcessGroupNCCL.cpp:669] [Rank 28] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.593144 21283 ProcessGroupNCCL.cpp:669] [Rank 38] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.593257 23453 ProcessGroupNCCL.cpp:835] [Rank 38] NCCL watchdog thread started! I1027 12:40:47.593163 21284 ProcessGroupNCCL.cpp:669] [Rank 39] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.593266 23454 ProcessGroupNCCL.cpp:835] [Rank 39] NCCL watchdog thread started! I1027 12:40:47.603930 6278 ProcessGroupNCCL.cpp:669] [Rank 54] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.593264 21280 ProcessGroupNCCL.cpp:669] [Rank 36] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.593394 23456 ProcessGroupNCCL.cpp:835] [Rank 36] NCCL watchdog thread started! I1027 12:40:47.604070 7980 ProcessGroupNCCL.cpp:835] [Rank 54] NCCL watchdog thread started! I1027 12:40:47.607358 22023 ProcessGroupNCCL.cpp:669] [Rank 29] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.593250 21282 ProcessGroupNCCL.cpp:669] [Rank 37] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.593345 23455 ProcessGroupNCCL.cpp:835] [Rank 37] NCCL watchdog thread started! I1027 12:40:47.604121 7981 ProcessGroupNCCL.cpp:835] [Rank 53] NCCL watchdog thread started! I1027 12:40:47.607376 23979 ProcessGroupNCCL.cpp:835] [Rank 28] NCCL watchdog thread started! I1027 12:40:47.605221 13436 ProcessGroupNCCL.cpp:669] [Rank 81] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.604033 6277 ProcessGroupNCCL.cpp:669] [Rank 53] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.607491 23980 ProcessGroupNCCL.cpp:835] [Rank 29] NCCL watchdog thread started! I1027 12:40:47.605345 15574 ProcessGroupNCCL.cpp:835] [Rank 81] NCCL watchdog thread started! I1027 12:40:47.607482 22025 ProcessGroupNCCL.cpp:669] [Rank 31] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.607614 23981 ProcessGroupNCCL.cpp:835] [Rank 31] NCCL watchdog thread started! I1027 12:40:47.605345 13437 ProcessGroupNCCL.cpp:669] [Rank 82] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.605480 15575 ProcessGroupNCCL.cpp:835] [Rank 82] NCCL watchdog thread started! I1027 12:40:47.605525 13438 ProcessGroupNCCL.cpp:669] [Rank 83] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.605607 15576 ProcessGroupNCCL.cpp:835] [Rank 83] NCCL watchdog thread started! I1027 12:40:47.603713 16939 ProcessGroupNCCL.cpp:669] [Rank 66] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.603826 18596 ProcessGroupNCCL.cpp:835] [Rank 66] NCCL watchdog thread started! I1027 12:40:47.605602 13434 ProcessGroupNCCL.cpp:669] [Rank 80] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.603767 16940 ProcessGroupNCCL.cpp:669] [Rank 67] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.603880 18597 ProcessGroupNCCL.cpp:835] [Rank 67] NCCL watchdog thread started! I1027 12:40:47.605712 15577 ProcessGroupNCCL.cpp:835] [Rank 80] NCCL watchdog thread started! I1027 12:40:47.604393 6275 ProcessGroupNCCL.cpp:669] [Rank 52] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.604516 7982 ProcessGroupNCCL.cpp:835] [Rank 52] NCCL watchdog thread started! I1027 12:40:47.604151 27005 ProcessGroupNCCL.cpp:669] [Rank 6] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.604243 29022 ProcessGroupNCCL.cpp:835] [Rank 6] NCCL watchdog thread started! I1027 12:40:47.604204 27006 ProcessGroupNCCL.cpp:669] [Rank 7] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.604293 29023 ProcessGroupNCCL.cpp:835] [Rank 7] NCCL watchdog thread started! I1027 12:40:47.604061 16936 ProcessGroupNCCL.cpp:669] [Rank 64] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.604326 27002 ProcessGroupNCCL.cpp:669] [Rank 4] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.604442 29024 ProcessGroupNCCL.cpp:835] [Rank 4] NCCL watchdog thread started! I1027 12:40:47.604169 18598 ProcessGroupNCCL.cpp:835] [Rank 64] NCCL watchdog thread started! I1027 12:40:47.604163 16938 ProcessGroupNCCL.cpp:669] [Rank 65] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.604444 27004 ProcessGroupNCCL.cpp:669] [Rank 5] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.604549 29025 ProcessGroupNCCL.cpp:835] [Rank 5] NCCL watchdog thread started! I1027 12:40:47.604223 18599 ProcessGroupNCCL.cpp:835] [Rank 65] NCCL watchdog thread started! I1027 12:40:47.591650 18788 ProcessGroupNCCL.cpp:669] [Rank 91] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.591754 20894 ProcessGroupNCCL.cpp:835] [Rank 91] NCCL watchdog thread started! I1027 12:40:47.591701 18784 ProcessGroupNCCL.cpp:669] [Rank 88] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.591815 20895 ProcessGroupNCCL.cpp:835] [Rank 88] NCCL watchdog thread started! I1027 12:40:47.601801 5142 ProcessGroupNCCL.cpp:669] [Rank 3] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.601907 7568 ProcessGroupNCCL.cpp:835] [Rank 3] NCCL watchdog thread started! I1027 12:40:47.601881 5140 ProcessGroupNCCL.cpp:669] [Rank 1] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.601991 7569 ProcessGroupNCCL.cpp:835] [Rank 1] NCCL watchdog thread started! I1027 12:40:47.591979 18786 ProcessGroupNCCL.cpp:669] [Rank 89] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.592089 20896 ProcessGroupNCCL.cpp:835] [Rank 89] NCCL watchdog thread started! I1027 12:40:47.601994 5136 ProcessGroupNCCL.cpp:669] [Rank 0] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.602084 7570 ProcessGroupNCCL.cpp:835] [Rank 0] NCCL watchdog thread started! I1027 12:40:47.592082 18787 ProcessGroupNCCL.cpp:669] [Rank 90] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.592207 20897 ProcessGroupNCCL.cpp:835] [Rank 90] NCCL watchdog thread started! I1027 12:40:47.602171 5141 ProcessGroupNCCL.cpp:669] [Rank 2] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.602284 7571 ProcessGroupNCCL.cpp:835] [Rank 2] NCCL watchdog thread started! I1027 12:40:47.606274 755 ProcessGroupNCCL.cpp:669] [Rank 43] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.606412 2963 ProcessGroupNCCL.cpp:835] [Rank 43] NCCL watchdog thread started! I1027 12:40:47.593822 24869 ProcessGroupNCCL.cpp:835] [Rank 61] NCCL watchdog thread started! I1027 12:40:47.593777 22737 ProcessGroupNCCL.cpp:669] [Rank 61] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.606343 751 ProcessGroupNCCL.cpp:669] [Rank 40] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.606465 2964 ProcessGroupNCCL.cpp:835] [Rank 40] NCCL watchdog thread started! I1027 12:40:47.606510 2965 ProcessGroupNCCL.cpp:835] [Rank 41] NCCL watchdog thread started! I1027 12:40:47.606462 753 ProcessGroupNCCL.cpp:669] [Rank 41] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.593806 874 ProcessGroupNCCL.cpp:669] [Rank 9] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.593858 876 ProcessGroupNCCL.cpp:669] [Rank 11] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.606590 754 ProcessGroupNCCL.cpp:669] [Rank 42] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.606714 2966 ProcessGroupNCCL.cpp:835] [Rank 42] NCCL watchdog thread started! I1027 12:40:47.593940 2868 ProcessGroupNCCL.cpp:835] [Rank 9] NCCL watchdog thread started! I1027 12:40:47.593981 2869 ProcessGroupNCCL.cpp:835] [Rank 11] NCCL watchdog thread started! I1027 12:40:47.594132 872 ProcessGroupNCCL.cpp:669] [Rank 8] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.607846 22284 ProcessGroupNCCL.cpp:669] [Rank 74] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.607937 24216 ProcessGroupNCCL.cpp:835] [Rank 74] NCCL watchdog thread started! I1027 12:40:47.607924 22285 ProcessGroupNCCL.cpp:669] [Rank 75] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.608031 24217 ProcessGroupNCCL.cpp:835] [Rank 75] NCCL watchdog thread started! I1027 12:40:47.608004 22283 ProcessGroupNCCL.cpp:669] [Rank 73] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.594221 2870 ProcessGroupNCCL.cpp:835] [Rank 8] NCCL watchdog thread started! I1027 12:40:47.608150 24218 ProcessGroupNCCL.cpp:835] [Rank 73] NCCL watchdog thread started! I1027 12:40:47.594159 875 ProcessGroupNCCL.cpp:669] [Rank 10] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.608054 22281 ProcessGroupNCCL.cpp:669] [Rank 72] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.594287 2871 ProcessGroupNCCL.cpp:835] [Rank 10] NCCL watchdog thread started! I1027 12:40:47.594194 22735 ProcessGroupNCCL.cpp:669] [Rank 60] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.608175 24219 ProcessGroupNCCL.cpp:835] [Rank 72] NCCL watchdog thread started! I1027 12:40:47.594310 22738 ProcessGroupNCCL.cpp:669] [Rank 62] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.594409 24871 ProcessGroupNCCL.cpp:835] [Rank 62] NCCL watchdog thread started! I1027 12:40:47.594298 24870 ProcessGroupNCCL.cpp:835] [Rank 60] NCCL watchdog thread started! I1027 12:40:47.594394 22739 ProcessGroupNCCL.cpp:669] [Rank 63] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.594514 24872 ProcessGroupNCCL.cpp:835] [Rank 63] NCCL watchdog thread started! I1027 12:40:47.607353 8671 ProcessGroupNCCL.cpp:669] [Rank 16] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.607460 10712 ProcessGroupNCCL.cpp:835] [Rank 16] NCCL watchdog thread started! I1027 12:40:47.607410 8675 ProcessGroupNCCL.cpp:669] [Rank 19] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.607558 10713 ProcessGroupNCCL.cpp:835] [Rank 19] NCCL watchdog thread started! I1027 12:40:47.607677 10715 ProcessGroupNCCL.cpp:835] [Rank 17] NCCL watchdog thread started! I1027 12:40:47.607620 8673 ProcessGroupNCCL.cpp:669] [Rank 17] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.607913 8674 ProcessGroupNCCL.cpp:669] [Rank 18] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.608012 10716 ProcessGroupNCCL.cpp:835] [Rank 18] NCCL watchdog thread started! I1027 12:40:47.608415 23994 ProcessGroupNCCL.cpp:669] [Rank 22] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.608672 26212 ProcessGroupNCCL.cpp:835] [Rank 20] NCCL watchdog thread started! I1027 12:40:47.608583 23991 ProcessGroupNCCL.cpp:669] [Rank 20] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.608525 26211 ProcessGroupNCCL.cpp:835] [Rank 22] NCCL watchdog thread started! I1027 12:40:47.608697 23995 ProcessGroupNCCL.cpp:669] [Rank 23] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.608798 26213 ProcessGroupNCCL.cpp:835] [Rank 23] NCCL watchdog thread started! I1027 12:40:47.608747 23993 ProcessGroupNCCL.cpp:669] [Rank 21] ProcessGroupNCCL initialized with following options: NCCL_ASYNC_ERROR_HANDLING: 0 NCCL_DESYNC_DEBUG: 0 NCCL_BLOCKING_WAIT: 0 TIMEOUT(ms): 1800000 USE_HIGH_PRIORITY_STREAM: 0 I1027 12:40:47.608842 26214 ProcessGroupNCCL.cpp:835] [Rank 21] NCCL watchdog thread started! 0%| | 0/420 [00:00