WARNING: Logging before flag parsing goes to stderr.
W0409 09:36:25.389571 47321472250304 deprecation.py:323] From /public/home/xuanbaby/rocm3.9-python3.6.8-tf1.15/lib/python3.6/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
2021-04-09 09:36:25.406578: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2000070000 Hz
2021-04-09 09:36:25.408573: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4d6b9f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-04-09 09:36:25.408704: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-04-09 09:36:25.411753: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libamdhip64.so
2021-04-09 09:36:25.560077: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4fef880 initialized for platform ROCM (this does not guarantee that XLA will be used). Devices:
2021-04-09 09:36:25.560217: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Device 66a1, AMDGPU ISA version: gfx906
2021-04-09 09:36:25.560254: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (1): Device 66a1, AMDGPU ISA version: gfx906
2021-04-09 09:36:25.560287: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (2): Device 66a1, AMDGPU ISA version: gfx906
2021-04-09 09:36:25.560331: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (3): Device 66a1, AMDGPU ISA version: gfx906
2021-04-09 09:36:27.162256: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties: 
pciBusID: 0000:04:00.0 name: Device 66a1     ROCm AMD GPU ISA: gfx906
coreClock: 1.7GHz coreCount: 64 deviceMemorySize: 15.98GiB deviceMemoryBandwidth: 953.67GiB/s
2021-04-09 09:36:27.162470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 1 with properties: 
pciBusID: 0000:26:00.0 name: Device 66a1     ROCm AMD GPU ISA: gfx906
coreClock: 1.7GHz coreCount: 64 deviceMemorySize: 15.98GiB deviceMemoryBandwidth: 953.67GiB/s
2021-04-09 09:36:27.162585: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 2 with properties: 
pciBusID: 0000:43:00.0 name: Device 66a1     ROCm AMD GPU ISA: gfx906
coreClock: 1.7GHz coreCount: 64 deviceMemorySize: 15.98GiB deviceMemoryBandwidth: 953.67GiB/s
2021-04-09 09:36:27.162696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 3 with properties: 
pciBusID: 0000:63:00.0 name: Device 66a1     ROCm AMD GPU ISA: gfx906
coreClock: 1.7GHz coreCount: 64 deviceMemorySize: 15.98GiB deviceMemoryBandwidth: 953.67GiB/s
2021-04-09 09:36:27.169566: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library librocblas.so
2021-04-09 09:36:27.173594: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libMIOpen.so
2021-04-09 09:36:27.195922: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library librocfft.so
2021-04-09 09:36:27.196755: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library librocrand.so
2021-04-09 09:36:27.197430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1, 2, 3
2021-04-09 09:36:27.197524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-09 09:36:27.197564: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 1 2 3 
2021-04-09 09:36:27.197596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N Y Y Y 
2021-04-09 09:36:27.197640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 1:   Y N Y Y 
2021-04-09 09:36:27.197671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 2:   Y Y N Y 
2021-04-09 09:36:27.197711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 3:   Y Y Y N 
2021-04-09 09:36:27.198262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15385 MB memory) -> physical GPU (device: 0, name: Device 66a1, pci bus id: 0000:04:00.0)
2021-04-09 09:36:27.203739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 15385 MB memory) -> physical GPU (device: 1, name: Device 66a1, pci bus id: 0000:26:00.0)
2021-04-09 09:36:27.209108: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 15385 MB memory) -> physical GPU (device: 2, name: Device 66a1, pci bus id: 0000:43:00.0)
2021-04-09 09:36:27.212895: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 15385 MB memory) -> physical GPU (device: 3, name: Device 66a1, pci bus id: 0000:63:00.0)
W0409 09:36:27.246325 47321472250304 deprecation.py:323] From /public/home/xuanbaby/DL-TensorFlow/benchmark-cnn_tf_v2.1_compatible/scripts/tf_cnn_benchmarks/convnet_builder.py:134: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0409 09:36:27.256055 47321472250304 deprecation.py:323] From /public/home/xuanbaby/rocm3.9-python3.6.8-tf1.15/lib/python3.6/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
W0409 09:36:27.295979 47321472250304 deprecation.py:323] From /public/home/xuanbaby/DL-TensorFlow/benchmark-cnn_tf_v2.1_compatible/scripts/tf_cnn_benchmarks/convnet_builder.py:266: max_pooling2d (from tensorflow.python.keras.legacy_tf_layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
W0409 09:36:30.278771 47321472250304 deprecation.py:323] From /public/home/xuanbaby/DL-TensorFlow/benchmark-cnn_tf_v2.1_compatible/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2268: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
W0409 09:36:30.763930 47321472250304 deprecation.py:323] From /public/home/xuanbaby/rocm3.9-python3.6.8-tf1.15/lib/python3.6/site-packages/tensorflow/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.tensor_shape_from_node_def_name`
2021-04-09 09:37:16.015443: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties: 
pciBusID: 0000:04:00.0 name: Device 66a1     ROCm AMD GPU ISA: gfx906
coreClock: 1.7GHz coreCount: 64 deviceMemorySize: 15.98GiB deviceMemoryBandwidth: 953.67GiB/s
2021-04-09 09:37:16.015732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 1 with properties: 
pciBusID: 0000:26:00.0 name: Device 66a1     ROCm AMD GPU ISA: gfx906
coreClock: 1.7GHz coreCount: 64 deviceMemorySize: 15.98GiB deviceMemoryBandwidth: 953.67GiB/s
2021-04-09 09:37:16.015851: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 2 with properties: 
pciBusID: 0000:43:00.0 name: Device 66a1     ROCm AMD GPU ISA: gfx906
coreClock: 1.7GHz coreCount: 64 deviceMemorySize: 15.98GiB deviceMemoryBandwidth: 953.67GiB/s
2021-04-09 09:37:16.015969: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 3 with properties: 
pciBusID: 0000:63:00.0 name: Device 66a1     ROCm AMD GPU ISA: gfx906
coreClock: 1.7GHz coreCount: 64 deviceMemorySize: 15.98GiB deviceMemoryBandwidth: 953.67GiB/s
2021-04-09 09:37:16.016111: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library librocblas.so
2021-04-09 09:37:16.016168: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libMIOpen.so
2021-04-09 09:37:16.016214: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library librocfft.so
2021-04-09 09:37:16.016259: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library librocrand.so
2021-04-09 09:37:16.016801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1, 2, 3
2021-04-09 09:37:16.016903: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-09 09:37:16.016944: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 1 2 3 
2021-04-09 09:37:16.016979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N Y Y Y 
2021-04-09 09:37:16.017010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 1:   Y N Y Y 
2021-04-09 09:37:16.017040: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 2:   Y Y N Y 
2021-04-09 09:37:16.017072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 3:   Y Y Y N 
2021-04-09 09:37:16.017472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15385 MB memory) -> physical GPU (device: 0, name: Device 66a1, pci bus id: 0000:04:00.0)
2021-04-09 09:37:16.017670: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 15385 MB memory) -> physical GPU (device: 1, name: Device 66a1, pci bus id: 0000:26:00.0)
2021-04-09 09:37:16.017862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 15385 MB memory) -> physical GPU (device: 2, name: Device 66a1, pci bus id: 0000:43:00.0)
2021-04-09 09:37:16.018043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 15385 MB memory) -> physical GPU (device: 3, name: Device 66a1, pci bus id: 0000:63:00.0)
2021-04-09 09:37:16.613226: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:508] ROCm Fusion is enabled.
108 ops no flops stats due to incomplete shapes.
I0409 09:37:16.865368 47321472250304 session_manager.py:505] Running local_init_op.
2021-04-09 09:37:16.896480: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:508] ROCm Fusion is enabled.
I0409 09:37:16.934625 47321472250304 session_manager.py:508] Done running local_init_op.
2021-04-09 09:37:17.156875: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:508] ROCm Fusion is enabled.
2021-04-09 09:37:17.201683: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:508] ROCm Fusion is enabled.
2021-04-09 09:37:17.223631: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:508] ROCm Fusion is enabled.
2021-04-09 09:37:18.191537: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:508] ROCm Fusion is enabled.
2021-04-09 09:37:18.283935: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library librocblas.so
2021-04-09 09:37:18.846362: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libMIOpen.so
2021-04-09 09:37:31.560452: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started.
2021-04-09 09:37:31.560664: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:945] Profiler found 4 GPUs
2021-04-09 09:37:31.560718: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:633] GpuTracer created.
2021-04-09 09:37:31.562417: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libroctracer64.so
2021-04-09 09:37:31.565194: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:965] GpuTracer started
2021-04-09 09:37:31.568309: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=137,) : Activity event encountered before a corresponding API event.
2021-04-09 09:37:31.577362: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=1052,) : Activity event encountered before a corresponding API event.
2021-04-09 09:37:31.582496: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=1133,) : Activity event encountered before a corresponding API event.
2021-04-09 09:37:31.598713: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=1680,) : Activity event encountered before a corresponding API event.
2021-04-09 09:37:31.652433: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=3883,) : Activity event encountered before a corresponding API event.
2021-04-09 09:37:31.652501: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=3902,) : Activity event encountered before a corresponding API event.
2021-04-09 09:37:31.652558: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=3919,) : Activity event encountered before a corresponding API event.
2021-04-09 09:37:31.667057: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=4739,) : Activity event encountered before a corresponding API event.
2021-04-09 09:37:31.835201: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=15803,) : Activity event encountered before a corresponding API event.
2021-04-09 09:37:31.861192: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=17259,) : Activity event encountered before a corresponding API event.
2021-04-09 09:37:31.861265: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=27001,) : Activity event encountered before a corresponding API event.
2021-04-09 09:37:31.861306: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=28385,) : Activity event encountered before a corresponding API event.
2021-04-09 09:37:31.861354: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=28391,) : Activity event encountered before a corresponding API event.
2021-04-09 09:37:31.862898: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:976] GpuTracer stopped
2021-04-09 09:38:18.045725: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started.
2021-04-09 09:38:18.046225: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:633] GpuTracer created.
2021-04-09 09:38:18.046923: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:965] GpuTracer started
2021-04-09 09:38:18.053424: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=28530,) : Activity event encountered before a corresponding API event.
2021-04-09 09:38:18.060946: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=28603,) : Activity event encountered before a corresponding API event.
2021-04-09 09:38:18.073171: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=29520,) : Activity event encountered before a corresponding API event.
2021-04-09 09:38:18.093675: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=30161,) : Activity event encountered before a corresponding API event.
2021-04-09 09:38:18.157205: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=32839,) : Activity event encountered before a corresponding API event.
2021-04-09 09:38:18.157378: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=32853,) : Activity event encountered before a corresponding API event.
2021-04-09 09:38:18.157425: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=32869,) : Activity event encountered before a corresponding API event.
2021-04-09 09:38:18.157845: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=32926,) : Activity event encountered before a corresponding API event.
2021-04-09 09:38:18.351695: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=45705,) : Activity event encountered before a corresponding API event.
2021-04-09 09:38:18.351769: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=46480,) : Activity event encountered before a corresponding API event.
2021-04-09 09:38:18.351811: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=57238,) : Activity event encountered before a corresponding API event.
2021-04-09 09:38:18.351849: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=57246,) : Activity event encountered before a corresponding API event.
2021-04-09 09:38:18.351904: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=57252,) : Activity event encountered before a corresponding API event.
2021-04-09 09:38:18.353202: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:976] GpuTracer stopped
2021-04-09 09:39:03.756964: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started.
2021-04-09 09:39:03.757193: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:633] GpuTracer created.
2021-04-09 09:39:03.757485: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:965] GpuTracer started
2021-04-09 09:39:03.764637: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=57391,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:03.776651: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=58306,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:03.781539: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=58385,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:03.797257: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=58907,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:03.844105: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=60931,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:03.844917: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=60946,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:03.844991: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=60967,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:03.866078: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=62165,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:03.991640: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=70820,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:04.059543: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=74533,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:04.059614: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=81463,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:04.060072: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=85223,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:04.060114: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=85229,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:04.062531: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:976] GpuTracer stopped
2021-04-09 09:39:49.077006: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started.
2021-04-09 09:39:49.077509: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:633] GpuTracer created.
2021-04-09 09:39:49.077730: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:965] GpuTracer started
2021-04-09 09:39:49.088948: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=85370,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:49.094427: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=85886,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:49.103019: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=86360,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:49.119206: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=86921,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:49.170822: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=89057,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:49.170923: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=89070,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:49.171116: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=89088,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:49.187296: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=89998,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:49.348073: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=100376,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:49.381125: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=102436,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:49.381190: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=111714,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:49.381230: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=113558,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:49.381267: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=113564,) : Activity event encountered before a corresponding API event.
2021-04-09 09:39:49.382882: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:976] GpuTracer stopped
2021-04-09 09:40:34.103760: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started.
2021-04-09 09:40:34.103959: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:633] GpuTracer created.
2021-04-09 09:40:34.104480: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:965] GpuTracer started
2021-04-09 09:40:34.112261: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=113703,) : Activity event encountered before a corresponding API event.
2021-04-09 09:40:34.115871: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=113855,) : Activity event encountered before a corresponding API event.
2021-04-09 09:40:34.126641: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=114691,) : Activity event encountered before a corresponding API event.
2021-04-09 09:40:34.142778: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=115263,) : Activity event encountered before a corresponding API event.
2021-04-09 09:40:34.194713: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=117413,) : Activity event encountered before a corresponding API event.
2021-04-09 09:40:34.198231: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=117530,) : Activity event encountered before a corresponding API event.
2021-04-09 09:40:34.198296: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=117552,) : Activity event encountered before a corresponding API event.
2021-04-09 09:40:34.211860: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=118313,) : Activity event encountered before a corresponding API event.
2021-04-09 09:40:34.373030: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=129241,) : Activity event encountered before a corresponding API event.
2021-04-09 09:40:34.404921: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=130997,) : Activity event encountered before a corresponding API event.
2021-04-09 09:40:34.404988: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=140145,) : Activity event encountered before a corresponding API event.
2021-04-09 09:40:34.405028: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=141891,) : Activity event encountered before a corresponding API event.
2021-04-09 09:40:34.405066: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=141899,) : Activity event encountered before a corresponding API event.
2021-04-09 09:40:34.406997: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:976] GpuTracer stopped
2021-04-09 09:41:18.810520: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started.
2021-04-09 09:41:18.811037: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:633] GpuTracer created.
2021-04-09 09:41:18.811277: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:965] GpuTracer started
2021-04-09 09:41:18.824301: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=142038,) : Activity event encountered before a corresponding API event.
2021-04-09 09:41:18.832722: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=142953,) : Activity event encountered before a corresponding API event.
2021-04-09 09:41:18.837837: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=143034,) : Activity event encountered before a corresponding API event.
2021-04-09 09:41:18.854114: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=143592,) : Activity event encountered before a corresponding API event.
2021-04-09 09:41:18.905605: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=145726,) : Activity event encountered before a corresponding API event.
2021-04-09 09:41:18.905686: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=145746,) : Activity event encountered before a corresponding API event.
2021-04-09 09:41:18.905905: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=145759,) : Activity event encountered before a corresponding API event.
2021-04-09 09:41:18.922990: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=146704,) : Activity event encountered before a corresponding API event.
2021-04-09 09:41:19.083335: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=157195,) : Activity event encountered before a corresponding API event.
2021-04-09 09:41:19.116029: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=159062,) : Activity event encountered before a corresponding API event.
2021-04-09 09:41:19.116094: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=168442,) : Activity event encountered before a corresponding API event.
2021-04-09 09:41:19.116135: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=170232,) : Activity event encountered before a corresponding API event.
2021-04-09 09:41:19.116172: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=170238,) : Activity event encountered before a corresponding API event.
2021-04-09 09:41:19.117573: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:976] GpuTracer stopped
2021-04-09 09:42:03.531082: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started.
2021-04-09 09:42:03.531621: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:633] GpuTracer created.
2021-04-09 09:42:03.531823: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:965] GpuTracer started
2021-04-09 09:42:03.539651: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=170377,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:03.545632: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=170722,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:03.554903: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=171367,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:03.571715: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=171940,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:03.632154: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=174344,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:03.632235: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=174370,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:03.632285: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=174394,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:03.639478: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=174835,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:03.832467: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=187550,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:03.832533: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=187755,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:03.832985: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=198977,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:03.833030: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=198983,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:03.833068: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=198989,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:03.834674: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:976] GpuTracer stopped
2021-04-09 09:42:48.170531: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started.
2021-04-09 09:42:48.170732: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:633] GpuTracer created.
2021-04-09 09:42:48.171237: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:965] GpuTracer started
2021-04-09 09:42:48.179715: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=199128,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:48.180245: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=199168,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:48.191949: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=200118,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:48.208896: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=200687,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:48.268787: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=203079,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:48.268863: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=203094,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:48.268910: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=203111,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:48.276889: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=203613,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:48.469716: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=216285,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:48.469786: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=216466,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:48.469827: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=227718,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:48.469915: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=227724,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:48.469959: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=227730,) : Activity event encountered before a corresponding API event.
2021-04-09 09:42:48.471447: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:976] GpuTracer stopped
2021-04-09 09:43:32.818393: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started.
2021-04-09 09:43:32.818598: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:633] GpuTracer created.
2021-04-09 09:43:32.819240: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:965] GpuTracer started
2021-04-09 09:43:32.826492: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=227869,) : Activity event encountered before a corresponding API event.
2021-04-09 09:43:32.827986: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=227999,) : Activity event encountered before a corresponding API event.
2021-04-09 09:43:32.840995: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=228857,) : Activity event encountered before a corresponding API event.
2021-04-09 09:43:32.857061: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=229427,) : Activity event encountered before a corresponding API event.
2021-04-09 09:43:32.909145: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=231589,) : Activity event encountered before a corresponding API event.
2021-04-09 09:43:32.910818: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=231622,) : Activity event encountered before a corresponding API event.
2021-04-09 09:43:32.910883: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=231644,) : Activity event encountered before a corresponding API event.
2021-04-09 09:43:32.925818: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=232477,) : Activity event encountered before a corresponding API event.
2021-04-09 09:43:33.085086: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=242964,) : Activity event encountered before a corresponding API event.
2021-04-09 09:43:33.119060: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=245135,) : Activity event encountered before a corresponding API event.
2021-04-09 09:43:33.119127: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=254137,) : Activity event encountered before a corresponding API event.
2021-04-09 09:43:33.119168: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=255987,) : Activity event encountered before a corresponding API event.
2021-04-09 09:43:33.119206: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=255995,) : Activity event encountered before a corresponding API event.
2021-04-09 09:43:33.121871: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:976] GpuTracer stopped
2021-04-09 09:44:17.502613: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started.
2021-04-09 09:44:17.503257: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:633] GpuTracer created.
2021-04-09 09:44:17.503473: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:965] GpuTracer started
2021-04-09 09:44:17.511689: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=256134,) : Activity event encountered before a corresponding API event.
2021-04-09 09:44:17.517226: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=256198,) : Activity event encountered before a corresponding API event.
2021-04-09 09:44:17.526744: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=257130,) : Activity event encountered before a corresponding API event.
2021-04-09 09:44:17.543732: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=257718,) : Activity event encountered before a corresponding API event.
2021-04-09 09:44:17.604481: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=260161,) : Activity event encountered before a corresponding API event.
2021-04-09 09:44:17.605714: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=260181,) : Activity event encountered before a corresponding API event.
2021-04-09 09:44:17.605766: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=260203,) : Activity event encountered before a corresponding API event.
2021-04-09 09:44:17.611490: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=260596,) : Activity event encountered before a corresponding API event.
2021-04-09 09:44:17.804412: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=273224,) : Activity event encountered before a corresponding API event.
2021-04-09 09:44:17.804594: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=273778,) : Activity event encountered before a corresponding API event.
2021-04-09 09:44:17.804864: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=284864,) : Activity event encountered before a corresponding API event.
2021-04-09 09:44:17.804907: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=284872,) : Activity event encountered before a corresponding API event.
2021-04-09 09:44:17.804945: I tensorflow/core/profiler/internal/gpu/device_tracer_rocm.cc:188] RocmTracerEvent dropped (correlation_id=284878,) : Activity event encountered before a corresponding API event.
2021-04-09 09:44:17.807091: I tensorflow/core/profiler/internal/gpu/rocm_tracer.cc:976] GpuTracer stopped

=========================Options=============================
-max_depth                  20
-min_bytes                  1
-min_peak_bytes             0
-min_residual_bytes         0
-min_output_bytes           0
-min_micros                 1
-min_accelerator_micros     0
-min_cpu_micros             0
-min_params                 0
-min_float_ops              0
-min_occurrence             0
-step                       -1
-order_by                   accelerator_micros
-account_type_regexes       .*
-start_name_regexes         .*
-trim_name_regexes          
-show_name_regexes          .*
-hide_name_regexes          
-account_displayed_op_only  true
-select                     bytes,micros
-output                     stdout:

==================Model Analysis Report======================

Doc:
op: The nodes are operation kernel type, such as MatMul, Conv2D. Graph nodes belonging to the same type are aggregated together.
requested bytes: The memory requested by the operation, accumulatively.
total execution time: Sum of accelerator execution time and cpu execution time.
cpu execution time: The time from the start to the end of the operation. It's the sum of actual cpu run time plus the time that it spends waiting if part of computation is launched asynchronously.
accelerator execution time: Time spent executing on the accelerator. This is normally measured by the actual hardware library.

Profile:
node name | requested bytes | total execution time | accelerator execution time | cpu execution time
ApplyGradientDescent                  0B (0.00%, 0.00%), 1263317485145.43sec (100.00%, 15.41%), 1263317485145.42sec (100.00%, 15.41%),       4.20ms (100.00%, 10.36%)
Conv2DBackpropInput         8396.65MB (100.00%, 49.16%), 922005292446.23sec (84.59%, 11.25%), 922005292446.23sec (84.59%, 11.25%),        6.49ms (89.64%, 16.00%)
Sub                                256B (50.84%, 0.00%), 782139633578.16sec (73.34%, 9.54%), 782139633578.16sec (73.34%, 9.54%),         2.26ms (73.64%, 5.57%)
Conv2D                       3180.51MB (50.84%, 18.62%), 778827712249.94sec (63.79%, 9.50%), 778827712249.94sec (63.79%, 9.50%),        6.22ms (68.07%, 15.35%)
AssignSub                             0B (0.00%, 0.00%), 743553595747.17sec (54.29%, 9.07%), 743553595747.17sec (54.29%, 9.07%),         2.45ms (52.72%, 6.03%)
Mul                                256B (32.21%, 0.00%), 742501279043.41sec (45.22%, 9.06%), 742501279043.41sec (45.22%, 9.06%),         2.43ms (46.69%, 5.99%)
L2Loss                          75.52KB (32.21%, 0.00%), 732978640834.05sec (36.16%, 8.94%), 732978640834.05sec (36.16%, 8.94%),         2.44ms (40.70%, 6.00%)
FusedBatchNormV3             2723.62MB (32.21%, 15.95%), 598983505565.33sec (27.22%, 7.31%), 598983505565.32sec (27.22%, 7.31%),        4.26ms (34.70%, 10.51%)
FusedBatchNormGradV3         2768.81MB (16.26%, 16.21%), 413628164496.88sec (19.91%, 5.05%), 413628164496.87sec (19.91%, 5.05%),         2.62ms (24.19%, 6.47%)
ReluGrad                              0B (0.00%, 0.00%), 376216677477.47sec (14.86%, 4.59%), 376216677477.47sec (14.86%, 4.59%),         1.14ms (17.72%, 2.80%)
Relu                                  0B (0.00%, 0.00%), 361291756439.33sec (10.27%, 4.41%), 361291756439.33sec (10.27%, 4.41%),         1.11ms (14.91%, 2.75%)
AddN                                  0B (0.00%, 0.00%), 172643371475.12sec (5.87%, 2.11%), 172643371475.12sec (5.87%, 2.11%),          543us (12.16%, 1.34%)
AddV2                                 0B (0.00%, 0.00%), 111290277537.27sec (3.76%, 1.36%), 111290277537.27sec (3.76%, 1.36%),          352us (10.82%, 0.87%)
SparseSoftmaxCrossEntropyWithLogits           768B (0.05%, 0.00%), 55217536823.01sec (2.40%, 0.67%), 55217536823.01sec (2.40%, 0.67%),           179us (9.96%, 0.44%)
MatMul                            8.98MB (0.05%, 0.05%), 34815727722.17sec (1.73%, 0.42%), 34815727722.17sec (1.73%, 0.42%),           226us (9.51%, 0.56%)
Greater                           1.02KB (0.00%, 0.00%), 29580250098.70sec (1.30%, 0.36%), 29580250098.70sec (1.30%, 0.36%),           635us (8.96%, 1.57%)
LessEqual                         1.28KB (0.00%, 0.00%), 25616797255.79sec (0.94%, 0.31%), 25616797255.79sec (0.94%, 0.31%),           175us (7.39%, 0.43%)
LogicalAnd                            0B (0.00%, 0.00%), 18840021040.26sec (0.63%, 0.23%), 18840021040.26sec (0.63%, 0.23%),            87us (6.96%, 0.21%)
RealDiv                               0B (0.00%, 0.00%), 16707656367.38sec (0.40%, 0.20%), 16707656367.38sec (0.40%, 0.20%),           258us (6.74%, 0.64%)
Cast                              1.54KB (0.00%, 0.00%), 16045324106.74sec (0.20%, 0.20%), 16045324106.74sec (0.20%, 0.20%),          2.48ms (6.11%, 6.11%)

======================End of Report==========================
TensorFlow:  2.3
Model:       resnet50
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  64 global
             64 per device
Num batches: 20
Num epochs:  0.00
Devices:     ['/gpu:0']
NUMA bind:   False
Data format: NCHW
Optimizer:   sgd
Variables:   parameter_server
==========
Generating training model
Initializing graph
Running warm up
Done warm up
Step	Img/sec	total_loss
1	images/sec: 173.5 +/- 0.0 (jitter = 0.0)	7.608
10	images/sec: 173.6 +/- 0.5 (jitter = 1.9)	7.849
20	images/sec: 196.3 +/- 5.9 (jitter = 40.0)	8.013
----------------------------------------------------------------
total images/sec: 2.82
----------------------------------------------------------------
Dumping ProfileProto to true
