Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
widedeep-tf-fp32.log 21.6 KB
Newer Older
sunzhq2's avatar
init  
sunzhq2 committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78 
INFO:LANUCH:******************* Pip Package Installing *******************
INFO:PerfEngine:******************* Backend Env Initization *******************
INFO:BackendStore:Loading Compile Backend: DCU
INFO:BackendStore:Loading Runtime Backend: DCU
INFO:PerfEngine:******************************************* Start to test model: widedeep-tf-fp32. *******************************************
INFO:PerfEngine:******************************************* Running Backend Compilation... *******************************************
INFO:PerfEngine:Running Backend Preoptimization...
INFO:DatasetStore:Loading Dataset: open_criteo_kaggle
INFO:CriteoKaggle:Initial...

  0%|          | 0/2000000 [00:00<?, ?it/s]
  1%|          | 15977/2000000 [00:00<00:12, 159759.11it/s]
  2%|▏         | 32807/2000000 [00:00<00:11, 164776.18it/s]
  2%|▏         | 49613/2000000 [00:00<00:11, 166269.68it/s]
  3%|▎         | 66523/2000000 [00:00<00:11, 167384.24it/s]
  4%|▍         | 83446/2000000 [00:00<00:11, 168044.95it/s]
  5%|▌         | 100290/2000000 [00:00<00:11, 168174.55it/s]
  6%|▌         | 117292/2000000 [00:00<00:11, 168772.10it/s]
  7%|▋         | 134264/2000000 [00:00<00:11, 169071.34it/s]
  8%|▊         | 151172/2000000 [00:00<00:10, 169004.06it/s]
  8%|▊         | 168073/2000000 [00:01<00:10, 168836.86it/s]
  9%|▉         | 184960/2000000 [00:01<00:10, 168844.46it/s]
 10%|█         | 201857/2000000 [00:01<00:10, 168878.39it/s]
 11%|█         | 218745/2000000 [00:01<00:10, 168793.51it/s]
 12%|█▏        | 235625/2000000 [00:01<00:10, 168584.29it/s]
 13%|█▎        | 252500/2000000 [00:01<00:10, 168632.41it/s]
 13%|█▎        | 269364/2000000 [00:01<00:10, 168400.03it/s]
 14%|█▍        | 286292/2000000 [00:01<00:10, 168660.87it/s]
 15%|█▌        | 303159/2000000 [00:01<00:10, 168309.20it/s]
 16%|█▌        | 320092/2000000 [00:01<00:09, 168612.61it/s]
 17%|█▋        | 336954/2000000 [00:02<00:09, 168139.98it/s]
 18%|█▊        | 353859/2000000 [00:02<00:09, 168408.98it/s]
 19%|█▊        | 370701/2000000 [00:02<00:09, 167604.23it/s]
 19%|█▉        | 387675/2000000 [00:02<00:09, 168239.77it/s]
 20%|██        | 404599/2000000 [00:02<00:09, 168536.39it/s]
 21%|██        | 421454/2000000 [00:02<00:09, 168154.12it/s]
 22%|██▏       | 438271/2000000 [00:02<00:09, 167648.53it/s]
 23%|██▎       | 455191/2000000 [00:02<00:09, 168109.95it/s]
 24%|██▎       | 472060/2000000 [00:02<00:09, 168279.16it/s]
 24%|██▍       | 488923/2000000 [00:02<00:08, 168382.67it/s]
 25%|██▌       | 505819/2000000 [00:03<00:08, 168552.64it/s]
 26%|██▌       | 522723/2000000 [00:03<00:08, 168695.73it/s]
 27%|██▋       | 539646/2000000 [00:03<00:08, 168852.43it/s]
 28%|██▊       | 556602/2000000 [00:03<00:08, 169061.25it/s]
 29%|██▊       | 573592/2000000 [00:03<00:08, 169310.16it/s]
 30%|██▉       | 590547/2000000 [00:03<00:08, 169377.78it/s]
 30%|███       | 607485/2000000 [00:03<00:08, 169340.35it/s]
 31%|███       | 624449/2000000 [00:03<00:08, 169428.45it/s]
 32%|███▏      | 641422/2000000 [00:03<00:08, 169512.64it/s]
 33%|███▎      | 658416/2000000 [00:03<00:07, 169636.63it/s]
 34%|███▍      | 675380/2000000 [00:04<00:07, 169591.60it/s]
 35%|███▍      | 692340/2000000 [00:04<00:07, 169503.14it/s]
 35%|███▌      | 709329/2000000 [00:04<00:07, 169616.27it/s]
 36%|███▋      | 726347/2000000 [00:04<00:07, 169782.70it/s]
 37%|███▋      | 743367/2000000 [00:04<00:07, 169903.77it/s]
 38%|███▊      | 760516/2000000 [00:04<00:07, 170375.56it/s]
 39%|███▉      | 777577/2000000 [00:04<00:07, 170444.10it/s]
 40%|███▉      | 794674/2000000 [00:04<00:07, 170600.19it/s]
 41%|████      | 811870/2000000 [00:04<00:06, 171005.08it/s]
 41%|████▏     | 829096/2000000 [00:04<00:06, 171377.41it/s]
 42%|████▏     | 846269/2000000 [00:05<00:06, 171479.55it/s]
 43%|████▎     | 863417/2000000 [00:05<00:06, 171358.28it/s]
 44%|████▍     | 880569/2000000 [00:05<00:06, 171404.98it/s]
 45%|████▍     | 897710/2000000 [00:05<00:06, 171402.86it/s]
 46%|████▌     | 914866/2000000 [00:05<00:06, 171446.11it/s]
 47%|████▋     | 932047/2000000 [00:05<00:06, 171553.46it/s]
 47%|████▋     | 949291/2000000 [00:05<00:06, 171817.22it/s]
 48%|████▊     | 966473/2000000 [00:05<00:06, 171791.32it/s]
 49%|████▉     | 983676/2000000 [00:05<00:05, 171859.64it/s]
 50%|█████     | 1000862/2000000 [00:05<00:05, 171486.73it/s]
 51%|█████     | 1018011/2000000 [00:06<00:05, 171083.01it/s]
 52%|█████▏    | 1035120/2000000 [00:06<00:05, 170741.11it/s]
 53%|█████▎    | 1052195/2000000 [00:06<00:05, 170478.41it/s]
 53%|█████▎    | 1069244/2000000 [00:06<00:05, 170460.78it/s]
 54%|█████▍    | 1086342/2000000 [00:06<00:05, 170613.77it/s]
 55%|█████▌    | 1103584/2000000 [00:06<00:05, 171150.70it/s]
 56%|█████▌    | 1120786/2000000 [00:06<00:05, 171408.89it/s]
 57%|█████▋    | 1138059/2000000 [00:06<00:05, 171803.32it/s]
 58%|█████▊    | 1155314/2000000 [00:06<00:04, 172023.55it/s]
 59%|█████▊    | 1172604/2000000 [00:06<00:04, 172284.57it/s]
 59%|█████▉    | 1189875/2000000 [00:07<00:04, 172410.35it/s]
 60%|██████    | 1207120/2000000 [00:07<00:04, 172420.20it/s]
 61%|██████    | 1224363/2000000 [00:07<00:04, 172182.22it/s]
 62%|██████▏   | 1241582/2000000 [00:07<00:04, 172129.26it/s]
 63%|██████▎   | 1258795/2000000 [00:07<00:04, 172105.55it/s]
 64%|██████▍   | 1276027/2000000 [00:07<00:04, 172167.67it/s]
 65%|██████▍   | 1293244/2000000 [00:07<00:04, 172088.09it/s]
 66%|██████▌   | 1310456/2000000 [00:07<00:04, 172095.86it/s]
 66%|██████▋   | 1327666/2000000 [00:07<00:03, 172049.55it/s]
 67%|██████▋   | 1344885/2000000 [00:07<00:03, 172088.51it/s]
 68%|██████▊   | 1362099/2000000 [00:08<00:03, 172102.03it/s]
 69%|██████▉   | 1379310/2000000 [00:08<00:03, 171705.32it/s]
 70%|██████▉   | 1396544/2000000 [00:08<00:03, 171893.58it/s]
 71%|███████   | 1413734/2000000 [00:08<00:03, 171875.67it/s]
 72%|███████▏  | 1430973/2000000 [00:08<00:03, 172026.42it/s]
 72%|███████▏  | 1448198/2000000 [00:08<00:03, 172091.53it/s]
 73%|███████▎  | 1465408/2000000 [00:08<00:03, 172029.31it/s]
 74%|███████▍  | 1482623/2000000 [00:08<00:03, 172063.82it/s]
 75%|███████▍  | 1499830/2000000 [00:08<00:02, 171969.58it/s]
 76%|███████▌  | 1517056/2000000 [00:08<00:02, 172051.77it/s]
 77%|███████▋  | 1534262/2000000 [00:09<00:02, 171984.82it/s]
 78%|███████▊  | 1551461/2000000 [00:09<00:02, 171648.40it/s]
 78%|███████▊  | 1568658/2000000 [00:09<00:02, 171742.91it/s]
 79%|███████▉  | 1585833/2000000 [00:09<00:02, 171738.22it/s]
 80%|████████  | 1603038/2000000 [00:09<00:02, 171828.64it/s]
 81%|████████  | 1620221/2000000 [00:09<00:02, 171824.06it/s]
 82%|████████▏ | 1637422/2000000 [00:09<00:02, 171875.81it/s]
 83%|████████▎ | 1654635/2000000 [00:09<00:02, 171948.65it/s]
 84%|████████▎ | 1671830/2000000 [00:09<00:01, 171913.33it/s]
 84%|████████▍ | 1689067/2000000 [00:09<00:01, 172046.52it/s]
 85%|████████▌ | 1706272/2000000 [00:10<00:01, 171902.81it/s]
 86%|████████▌ | 1723475/2000000 [00:10<00:01, 171938.77it/s]
 87%|████████▋ | 1740669/2000000 [00:10<00:01, 171571.61it/s]
 88%|████████▊ | 1757851/2000000 [00:10<00:01, 171643.83it/s]
 89%|████████▉ | 1775100/2000000 [00:10<00:01, 171893.10it/s]
 90%|████████▉ | 1792307/2000000 [00:10<00:01, 171942.85it/s]
 90%|█████████ | 1809529/2000000 [00:10<00:01, 172024.39it/s]
 91%|█████████▏| 1826766/2000000 [00:10<00:01, 172126.23it/s]
 92%|█████████▏| 1843979/2000000 [00:10<00:00, 172064.05it/s]
 93%|█████████▎| 1861226/2000000 [00:10<00:00, 172183.85it/s]
 94%|█████████▍| 1878445/2000000 [00:11<00:00, 172154.48it/s]
 95%|█████████▍| 1895667/2000000 [00:11<00:00, 172170.24it/s]
 96%|█████████▌| 1912886/2000000 [00:11<00:00, 172172.77it/s]
 97%|█████████▋| 1930106/2000000 [00:11<00:00, 172177.54it/s]
 97%|█████████▋| 1947362/2000000 [00:11<00:00, 172287.63it/s]
 98%|█████████▊| 1964591/2000000 [00:11<00:00, 171562.40it/s]
 99%|█████████▉| 1981806/2000000 [00:11<00:00, 171733.78it/s]
100%|█████████▉| 1999034/2000000 [00:11<00:00, 171892.44it/s]
100%|██████████| 2000000/2000000 [00:11<00:00, 170624.28it/s]
INFO:CriteoKaggle:Preprocessing...
INFO:CriteoKaggle:Rebatching batch size to: 1 ...

  0%|          | 0/2000000 [00:00<?, ?it/s]
  3%|▎         | 61843/2000000 [00:00<00:03, 618408.47it/s]
  6%|▋         | 125116/2000000 [00:00<00:02, 626814.67it/s]
  9%|▉         | 188320/2000000 [00:00<00:02, 629190.61it/s]
 13%|█▎        | 252054/2000000 [00:00<00:02, 632397.34it/s]
 16%|█▌        | 315634/2000000 [00:00<00:02, 633615.29it/s]
 19%|█▉        | 379097/2000000 [00:00<00:02, 633954.32it/s]
 22%|██▏       | 442852/2000000 [00:00<00:02, 635122.08it/s]
 25%|██▌       | 506526/2000000 [00:00<00:02, 635630.58it/s]
 29%|██▊       | 570168/2000000 [00:00<00:02, 635873.28it/s]
 32%|███▏      | 633756/2000000 [00:01<00:02, 635842.78it/s]
 35%|███▍      | 697341/2000000 [00:01<00:02, 635324.60it/s]
 38%|███▊      | 760874/2000000 [00:01<00:01, 635153.77it/s]
 41%|████▏     | 825028/2000000 [00:01<00:01, 637081.41it/s]
 44%|████▍     | 888737/2000000 [00:01<00:01, 636418.55it/s]
 48%|████▊     | 952380/2000000 [00:01<00:01, 635150.42it/s]
 51%|█████     | 1016598/2000000 [00:01<00:01, 637256.55it/s]
 54%|█████▍    | 1080325/2000000 [00:01<00:01, 635520.62it/s]
 57%|█████▋    | 1143879/2000000 [00:01<00:01, 633349.63it/s]
 60%|██████    | 1207581/2000000 [00:01<00:01, 634440.92it/s]
 64%|██████▎   | 1271589/2000000 [00:02<00:01, 636123.43it/s]
 67%|██████▋   | 1335740/2000000 [00:02<00:01, 637728.73it/s]
 70%|██████▉   | 1399515/2000000 [00:02<00:00, 637275.90it/s]
 73%|███████▎  | 1463637/2000000 [00:02<00:00, 638451.83it/s]
 76%|███████▋  | 1527484/2000000 [00:02<00:00, 635238.59it/s]
 80%|███████▉  | 1591675/2000000 [00:02<00:00, 637224.11it/s]
 83%|████████▎ | 1655495/2000000 [00:02<00:00, 637510.29it/s]
 86%|████████▌ | 1719249/2000000 [00:02<00:00, 632980.06it/s]
 89%|████████▉ | 1783454/2000000 [00:02<00:00, 635679.64it/s]
 92%|█████████▏| 1847742/2000000 [00:02<00:00, 637825.62it/s]
 96%|█████████▌| 1911531/2000000 [00:03<00:00, 633417.84it/s]
 99%|█████████▉| 1975652/2000000 [00:03<00:00, 635735.04it/s]
100%|██████████| 2000000/2000000 [00:03<00:00, 635265.12it/s]
INFO:PerfEngine:Start to compile the model...
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:absl:Fingerprint not found. Saved model loading will continue.
INFO:PerfEngine:******************************************* Running Accuracy Checker... *******************************************
INFO:CriteoKaggle:Rebatching batch size to: 1024 ...

  0%|          | 0/1953 [00:00<?, ?it/s]
100%|██████████| 1953/1953 [00:00<00:00, 602358.68it/s]
INFO:TestAccuracy:Start to calculate accuracy...

  0%|          | 0/1953 [00:00<?, ?it/s]WARNING:absl:`new_categorical_placeholder:0` is not a valid tf.function parameter name. Sanitizing to `new_categorical_placeholder_0`.
WARNING:absl:`new_numeric_placeholder:0` is not a valid tf.function parameter name. Sanitizing to `new_numeric_placeholder_0`.

  0%|          | 1/1953 [00:03<2:06:35,  3.89s/it]WARNING:absl:`new_categorical_placeholder:0` is not a valid tf.function parameter name. Sanitizing to `new_categorical_placeholder_0`.
WARNING:absl:`new_numeric_placeholder:0` is not a valid tf.function parameter name. Sanitizing to `new_numeric_placeholder_0`.
WARNING:absl:`new_categorical_placeholder:0` is not a valid tf.function parameter name. Sanitizing to `new_categorical_placeholder_0`.

  1%|▏         | 26/1953 [00:03<03:31,  9.10it/s] 
  3%|▎         | 52/1953 [00:04<01:29, 21.13it/s]
  4%|▍         | 78/1953 [00:04<00:51, 36.39it/s]
  5%|▌         | 104/1953 [00:04<00:33, 54.99it/s]
  7%|▋         | 130/1953 [00:04<00:23, 76.61it/s]
  8%|▊         | 156/1953 [00:04<00:17, 100.36it/s]
  9%|▉         | 182/1953 [00:04<00:14, 124.93it/s]
 11%|█         | 207/1953 [00:04<00:11, 147.84it/s]
 12%|█▏        | 233/1953 [00:04<00:10, 169.98it/s]
 13%|█▎        | 259/1953 [00:04<00:08, 188.93it/s]
 15%|█▍        | 285/1953 [00:05<00:08, 204.53it/s]
 16%|█▌        | 311/1953 [00:05<00:07, 216.97it/s]
 17%|█▋        | 337/1953 [00:05<00:07, 226.36it/s]
 19%|█▊        | 363/1953 [00:05<00:06, 233.37it/s]
 20%|█▉        | 389/1953 [00:05<00:06, 238.50it/s]
 21%|██        | 415/1953 [00:05<00:06, 242.38it/s]
 23%|██▎       | 441/1953 [00:05<00:06, 245.00it/s]
 24%|██▍       | 467/1953 [00:05<00:06, 247.06it/s]
 25%|██▌       | 493/1953 [00:05<00:05, 248.31it/s]
 27%|██▋       | 519/1953 [00:05<00:05, 249.46it/s]
 28%|██▊       | 545/1953 [00:06<00:05, 250.19it/s]
 29%|██▉       | 571/1953 [00:06<00:05, 250.75it/s]
 31%|███       | 597/1953 [00:06<00:05, 251.05it/s]
 32%|███▏      | 623/1953 [00:06<00:05, 251.13it/s]
 33%|███▎      | 649/1953 [00:06<00:05, 251.30it/s]
 35%|███▍      | 675/1953 [00:06<00:05, 249.35it/s]
 36%|███▌      | 700/1953 [00:06<00:05, 248.66it/s]
 37%|███▋      | 725/1953 [00:06<00:04, 248.13it/s]
 38%|███▊      | 750/1953 [00:06<00:04, 247.67it/s]
 40%|███▉      | 775/1953 [00:06<00:04, 247.52it/s]
 41%|████      | 800/1953 [00:07<00:04, 247.44it/s]
 42%|████▏     | 825/1953 [00:07<00:04, 247.39it/s]
 44%|████▎     | 850/1953 [00:07<00:04, 247.23it/s]
 45%|████▍     | 875/1953 [00:07<00:04, 247.13it/s]
 46%|████▌     | 900/1953 [00:07<00:04, 247.03it/s]
 47%|████▋     | 925/1953 [00:07<00:04, 247.06it/s]
 49%|████▊     | 950/1953 [00:07<00:04, 246.99it/s]
 50%|████▉     | 975/1953 [00:07<00:03, 247.03it/s]
 51%|█████     | 1000/1953 [00:07<00:03, 246.44it/s]
 52%|█████▏    | 1025/1953 [00:07<00:03, 246.36it/s]
 54%|█████▍    | 1050/1953 [00:08<00:03, 246.61it/s]
 55%|█████▌    | 1075/1953 [00:08<00:03, 246.89it/s]
 56%|█████▋    | 1100/1953 [00:08<00:03, 247.02it/s]
 58%|█████▊    | 1125/1953 [00:08<00:03, 246.94it/s]
 59%|█████▉    | 1150/1953 [00:08<00:03, 247.01it/s]
 60%|██████    | 1175/1953 [00:08<00:03, 247.04it/s]
 61%|██████▏   | 1200/1953 [00:08<00:03, 247.10it/s]
 63%|██████▎   | 1225/1953 [00:08<00:02, 247.14it/s]
 64%|██████▍   | 1250/1953 [00:08<00:02, 244.76it/s]
 65%|██████▌   | 1276/1953 [00:09<00:02, 246.58it/s]
 67%|██████▋   | 1301/1953 [00:09<00:02, 247.58it/s]
 68%|██████▊   | 1327/1953 [00:09<00:02, 248.48it/s]
 69%|██████▉   | 1352/1953 [00:09<00:02, 247.68it/s]
 71%|███████   | 1377/1953 [00:09<00:02, 248.31it/s]
 72%|███████▏  | 1402/1953 [00:09<00:02, 248.77it/s]
 73%|███████▎  | 1427/1953 [00:09<00:02, 249.01it/s]
 74%|███████▍  | 1453/1953 [00:09<00:02, 249.39it/s]
 76%|███████▌  | 1479/1953 [00:09<00:01, 249.57it/s]
 77%|███████▋  | 1504/1953 [00:09<00:01, 249.64it/s]
 78%|███████▊  | 1530/1953 [00:10<00:01, 249.98it/s]
 80%|███████▉  | 1555/1953 [00:10<00:01, 249.94it/s]
 81%|████████  | 1581/1953 [00:10<00:01, 250.12it/s]
 82%|████████▏ | 1607/1953 [00:10<00:01, 250.42it/s]
 84%|████████▎ | 1633/1953 [00:10<00:01, 250.71it/s]
 85%|████████▍ | 1659/1953 [00:10<00:01, 250.86it/s]
 86%|████████▋ | 1685/1953 [00:10<00:01, 250.90it/s]
 88%|████████▊ | 1711/1953 [00:10<00:00, 251.08it/s]
 89%|████████▉ | 1737/1953 [00:10<00:00, 251.14it/s]
 90%|█████████ | 1763/1953 [00:10<00:00, 251.19it/s]
 92%|█████████▏| 1789/1953 [00:11<00:00, 251.07it/s]
 93%|█████████▎| 1815/1953 [00:11<00:00, 251.10it/s]
 94%|█████████▍| 1841/1953 [00:11<00:00, 251.01it/s]
 96%|█████████▌| 1867/1953 [00:11<00:00, 250.98it/s]
 97%|█████████▋| 1893/1953 [00:11<00:00, 250.87it/s]
 98%|█████████▊| 1919/1953 [00:11<00:00, 250.75it/s]
100%|█████████▉| 1945/1953 [00:11<00:00, 250.76it/s]
100%|██████████| 1953/1953 [00:11<00:00, 166.71it/s]
INFO:TestAccuracy:Batch size is 1024, Accuracy: 0.77394
INFO:PerfEngine:******************************************* Runing QPS Checker... *******************************************
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:absl:Fingerprint not found. Saved model loading will continue.
INFO:BackendDCU:Batch size is 1024, QPS: 994174, Avg Latency:1.03, Tail Latency:1.39
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:absl:Fingerprint not found. Saved model loading will continue.
INFO:BackendDCU:Batch size is 20000, QPS: 2869440, Avg Latency:6.97, Tail Latency:7.05
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:absl:Fingerprint not found. Saved model loading will continue.
INFO:BackendDCU:Batch size is 40000, QPS: 2941176, Avg Latency:13.6, Tail Latency:16.03
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:absl:Fingerprint not found. Saved model loading will continue.
INFO:BackendDCU:Batch size is 80000, QPS: 2881844, Avg Latency:27.76, Tail Latency:30.16
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:absl:Fingerprint not found. Saved model loading will continue.
INFO:BackendDCU:Batch size is 120000, QPS: 3425635, Avg Latency:35.03, Tail Latency:37.36
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:absl:Fingerprint not found. Saved model loading will continue.
INFO:BackendDCU:Batch size is 140000, QPS: 3406326, Avg Latency:41.1, Tail Latency:43.22
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:absl:Fingerprint not found. Saved model loading will continue.
INFO:BackendDCU:Batch size is 160000, QPS: 3436426, Avg Latency:46.56, Tail Latency:48.9
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:absl:Fingerprint not found. Saved model loading will continue.
INFO:BackendDCU:Batch size is 180000, QPS: 2748511, Avg Latency:65.49, Tail Latency:67.9
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:absl:Fingerprint not found. Saved model loading will continue.
INFO:BackendDCU:Batch size is 200000, QPS: 3251503, Avg Latency:61.51, Tail Latency:63.47
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:absl:Fingerprint not found. Saved model loading will continue.
INFO:BackendDCU:Batch size is 220000, QPS: 2696078, Avg Latency:81.6, Tail Latency:83.84
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:absl:Fingerprint not found. Saved model loading will continue.
INFO:BackendDCU:Batch size is 240000, QPS: 2652226, Avg Latency:90.49, Tail Latency:92.63
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
INFO:absl:Fingerprint not found. Saved model loading will continue.
INFO:BackendDCU:Batch size is 260000, QPS: 2758913, Avg Latency:94.24, Tail Latency:96.49
INFO:PerfEngine:Testing Finish. Report is saved in path: [ general_perf/reports/DCU/widedeep-tf-fp32/result-fp32.json ]
INFO:PerfEngine:PDF Version is saved in path: [ general_perf/reports/DCU/widedeep-tf-fp32/WIDEDEEP-TF-FP32-TO-FP32.JSON.pdf ]
Exception ignored in: <function AtomicFunction.__del__ at 0x7f7ce8b5ca60>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py", line 218, in __del__
TypeError: 'NoneType' object is not subscriptable
Exception ignored in: <function AtomicFunction.__del__ at 0x7f7ce8b5ca60>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py", line 218, in __del__
TypeError: 'NoneType' object is not subscriptable
Exception ignored in: <function AtomicFunction.__del__ at 0x7f7ce8b5ca60>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py", line 218, in __del__
TypeError: 'NoneType' object is not subscriptable