使用东方系统跑多节点训练报错
将docker镜像封装成singularity镜像，安装必要的依赖包后，在东方系统使用salloc分配两个节点，每个节点运行yolov5训练代码，头节点报错如下 Starting training for 1000 epochs...
Epoch gpu_mem box obj cls labels img_size
0/999 1.37G 0.1164 0.04871 0.06597 489 640: 0%|▏ | 1/463 [00:13<1:43:35, 13.45s/it] Reducer buckets have been rebuilt in this iteration.
0/999 12.9G 0.09416 0.05832 0.05117 444 640: 100%|███████████████████████████████████████████████████████████████████████████████████████████▊| 462/463 [06:59<00:00, 1.16it/s] MIOpen(HIP): Error [LoadProgram] /data/jenkins_workspace/workspace/miopen_release_1/miopen/src/sqlite_db.cpp:254: Timeout while waiting for Database: /public/home/systest/.cache/miopen/2.15.2.2dbc7f85e/g fx906_64.ukdb. rc=00000005MIOpen(HIP): Error [LoadProgram] /data/jenkins_workspace/workspace/miopen_release_1/miopen/src/sqlite_db.cpp:254: Timeout while waiting for Database: /public/home/systest/.cache/miopen/2.15.2.2dbc7f85e/g fx906_64.ukdb. rc=00000005 MIOpen(HIP): Error [LoadProgram] /data/jenkins_workspace/workspace/miopen_release_1/miopen/src/sqlite_db.cpp:254: Timeout while waiting for Database: /public/home/systest/.cache/miopen/2.15.2.2dbc7f85e/g fx906_64.ukdb. rc=00000005 MIOpen(HIP): Error [LoadProgram] /data/jenkins_workspace/workspace/miopen_release_1/miopen/src/sqlite_db.cpp:254: Timeout while waiting for Database: /public/home/systest/.cache/miopen/2.15.2.2dbc7f85e/g fx906_64.ukdb. rc=00000005 MIOpen(HIP): Error [LoadProgram] /data/jenkins_workspace/workspace/miopen_release_1/miopen/src/sqlite_db.cpp:254: Timeout while waiting for Database: /public/home/systest/.cache/miopen/2.15.2.2dbc7f85e/g fx906_64.ukdb. rc=00000005 MIOpen(HIP): Error [LoadProgram] /data/jenkins_workspace/workspace/miopen_release_1/miopen/src/sqlite_db.cpp:254: Timeout while waiting for Database: /public/home/systest/.cache/miopen/2.15.2.2dbc7f85e/g fx906_64.ukdb. rc=00000005 MIOpen(HIP): Error [LoadProgram] /data/jenkins_workspace/workspace/miopen_release_1/miopen/src/sqlite_db.cpp:254: Timeout while waiting for Database: /public/home/systest/.cache/miopen/2.15.2.2dbc7f85e/g fx906_64.ukdb. rc=00000005 MIOpen(HIP): Error [LoadProgram] /data/jenkins_workspace/workspace/miopen_release_1/miopen/src/sqlite_db.cpp:254: Timeout while waiting for Database: /public/home/systest/.cache/miopen/2.15.2.2dbc7f85e/g fx906_64.ukdb. rc=00000005 MIOpen(HIP): Error [LoadProgram] /data/jenkins_workspace/workspace/miopen_release_1/miopen/src/sqlite_db.cpp:254: Timeout while waiting for Database: /public/home/systest/.cache/miopen/2.15.2.2dbc7f85e/g fx906_64.ukdb. rc=00000005 MIOpen(HIP): Error [LoadProgram] /data/jenkins_workspace/workspace/miopen_release_1/miopen/src/sqlite_db.cpp:254: Timeout while waiting for Database: /public/home/systest/.cache/miopen/2.15.2.2dbc7f85e/g fx906_64.ukdb. rc=00000005 MIOpen(HIP): Error [LoadProgram] /data/jenkins_workspace/workspace/miopen_release_1/miopen/src/sqlite_db.cpp:254: Timeout while waiting for Database: /public/home/systest/.cache/miopen/2.15.2.2dbc7f85e/g fx906_64.ukdb. rc=00000005 MIOpen(HIP): Error [LoadProgram] /data/jenkins_workspace/workspace/miopen_release_1/miopen/src/sqlite_db.cpp:254: Timeout while waiting for Database: /public/home/systest/.cache/miopen/2.15.2.2dbc7f85e/g fx906_64.ukdb. rc=00000005 MIOpen(HIP): Error [LoadProgram] /data/jenkins_workspace/workspace/miopen_release_1/miopen/src/sqlite_db.cpp:254: Timeout while waiting for Database: /public/home/systest/.cache/miopen/2.15.2.2dbc7f85e/g fx906_64.ukdb. rc=00000005 0/999 0.707G 0.09411 0.05832 0.05118 20 640: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 463/463 [13:40<00:00, 1.77s/it] Class Images Labels P R mAP@.5 mAP@.5:.95: 100%|██████████████████████████████████████████████████████████████████████████████| 79/79 [01:02<00:00, 1.25it/s] all 5000 36335 0.0041 0.0574 0.00351 0.0012