深度学习训练时间过长，GPU显存占用很多但是占用率过低问题

配置环境

操作系统：Ubuntu20.04
CUDA版本：10.2
Pytorch版本：1.6.0
TorchVision版本：0.7.0
mmdet版本：2.5.0
mmcv版本：1.1.5
IDE：PyCharm
硬件：RTX2070S*2

问题描述

在训练YOLOv4tiny时发现GPU占用率非常低，并且经常跳到0，导致训练速度很慢

为此博主对几个时间点就行设置，打印出来加载数据花费的时间和真正网络训练花费的时间，结果加载数据花费了20多秒，训练也只20多秒。加载数据出大问题，加载数据本文用的是Pytorch中自带的DataLoader

 gen = DataLoader(train_dataset, shuffle=True, batch_size=Batch_size, num_workers=32, pin_memory=True,
                             drop_last=True, collate_fn=yolo_dataset_collate)

调参结果

为了弄清楚要调什么参数来加快加速训练，做了以下实验

修改num_workers

截取日志中一部分进行日志说明：

num_workers=32, pin_memory=True
############################################
**************************************************
0.00020313262939453125
Epoch 1/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 23.64951467514038
**************************************************
every batch cost: 2.750419855117798
Epoch 1/50:   5%|▌         | 2/39 [00:27<11:37, 18.84s/it, lr=0.001, step/s=1.21, 

......

Epoch 1/50: 100%|██████████| 39/39 [00:58<00:00,  1.51s/it, lr=0.001, step/s=0.623, total_loss=406]

############################################上的为此时的参数，这次实验便是num_workers=32, pin_memory=True
在每一次加载数据后都会将加载时间写在第一个Epoch下，此次实验便是23.64951467514038s
后面的every batch cost: 2.750419855117798为网络真正训练一步花费的时间
当出现100%|██████████| 39/39 [00:58<00:00,时，表示一个epoch训练结束，总花费时间为58s（包含加载数据时间）
结果见附录：

结论

随着num_workers的减少加载数据花费的时间反而少了，但是整个训练段时间却加长了
最佳的num_workers应为32

附录

num_workers=32, pin_memory=True
############################################
**************************************************
0.00020313262939453125
Epoch 1/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 23.64951467514038
**************************************************
every batch cost: 2.750419855117798
Epoch 1/50:   5%|▌         | 2/39 [00:27<11:37, 18.84s/it, lr=0.001, step/s=1.21, total_loss=1.66e+3]**************************************************
every batch cost: 1.2042901515960693
Epoch 1/50:   8%|▊         | 3/39 [00:28<08:08, 13.57s/it, lr=0.001, step/s=1.26, total_loss=1.57e+3]**************************************************
every batch cost: 1.2558915615081787
Epoch 1/50:  10%|█         | 4/39 [00:34<06:32, 11.20s/it, lr=0.001, step/s=5.67, total_loss=1.48e+3]**************************************************
every batch cost: 1.228856086730957
Epoch 1/50:  13%|█▎        | 5/39 [00:35<04:37,  8.17s/it, lr=0.001, step/s=1.09, total_loss=1.4e+3]**************************************************
every batch cost: 1.0908327102661133
**************************************************
every batch cost: 0.7014598846435547
Epoch 1/50:  18%|█▊        | 7/39 [00:38<02:33,  4.79s/it, lr=0.001, step/s=1.05, total_loss=1.26e+3]**************************************************
every batch cost: 1.0383291244506836
Epoch 1/50:  21%|██        | 8/39 [00:39<01:53,  3.66s/it, lr=0.001, step/s=1.03, total_loss=1.19e+3]**************************************************
every batch cost: 0.8855016231536865
Epoch 1/50:  23%|██▎       | 9/39 [00:41<01:27,  2.90s/it, lr=0.001, step/s=1.11, total_loss=1.13e+3]**************************************************
every batch cost: 1.1094863414764404
Epoch 1/50:  26%|██▌       | 10/39 [00:42<01:07,  2.34s/it, lr=0.001, step/s=1.02, total_loss=1.07e+3]**************************************************
every batch cost: 1.0229365825653076
**************************************************
every batch cost: 1.1253528594970703
Epoch 1/50:  31%|███       | 12/39 [00:44<00:45,  1.70s/it, lr=0.001, step/s=1.03, total_loss=974]**************************************************
every batch cost: 1.0269100666046143
Epoch 1/50:  33%|███▎      | 13/39 [00:44<00:34,  1.33s/it, lr=0.001, step/s=0.464, total_loss=930]**************************************************
every batch cost: 0.46166372299194336
**************************************************
every batch cost: 0.442338228225708
Epoch 1/50:  38%|███▊      | 15/39 [00:45<00:21,  1.09it/s, lr=0.001, step/s=0.538, total_loss=851]**************************************************
every batch cost: 0.5364859104156494
Epoch 1/50:  41%|████      | 16/39 [00:46<00:18,  1.27it/s, lr=0.001, step/s=0.486, total_loss=816]**************************************************
every batch cost: 0.484661340713501
Epoch 1/50:  44%|████▎     | 17/39 [00:46<00:15,  1.45it/s, lr=0.001, step/s=0.455, total_loss=783]**************************************************
every batch cost: 0.45422983169555664
**************************************************
every batch cost: 0.46462392807006836
Epoch 1/50:  49%|████▊     | 19/39 [00:47<00:12,  1.59it/s, lr=0.001, step/s=0.632, total_loss=724]**************************************************
every batch cost: 0.6317436695098877
**************************************************
every batch cost: 0.4574618339538574
Epoch 1/50:  54%|█████▍    | 21/39 [00:48<00:09,  1.81it/s, lr=0.001, step/s=0.48, total_loss=672]**************************************************
every batch cost: 0.4788329601287842
**************************************************
every batch cost: 0.6374142169952393
Epoch 1/50:  59%|█████▉    | 23/39 [00:49<00:08,  1.82it/s, lr=0.001, step/s=0.466, total_loss=627]**************************************************
every batch cost: 0.4624333381652832
Epoch 1/50:  62%|██████▏   | 24/39 [00:50<00:07,  1.88it/s, lr=0.001, step/s=0.478, total_loss=607]**************************************************
every batch cost: 0.4764266014099121
**************************************************
every batch cost: 0.48482751846313477
Epoch 1/50:  67%|██████▋   | 26/39 [00:51<00:07,  1.85it/s, lr=0.001, step/s=0.582, total_loss=569]**************************************************
every batch cost: 0.5818459987640381
Epoch 1/50:  69%|██████▉   | 27/39 [00:52<00:06,  1.79it/s, lr=0.001, step/s=0.595, total_loss=552]**************************************************
every batch cost: 0.591942548751831
**************************************************
every batch cost: 0.45967578887939453
Epoch 1/50:  74%|███████▍  | 29/39 [00:53<00:05,  1.99it/s, lr=0.001, step/s=0.421, total_loss=521]**************************************************
every batch cost: 0.4171140193939209
Epoch 1/50:  77%|███████▋  | 30/39 [00:53<00:04,  1.89it/s, lr=0.001, step/s=0.584, total_loss=507]**************************************************
every batch cost: 0.5828967094421387
Epoch 1/50:  79%|███████▉  | 31/39 [00:54<00:04,  1.93it/s, lr=0.001, step/s=0.48, total_loss=493]**************************************************
every batch cost: 0.4693794250488281
**************************************************
every batch cost: 0.47809910774230957
Epoch 1/50:  85%|████████▍ | 33/39 [00:55<00:02,  2.00it/s, lr=0.001, step/s=0.468, total_loss=468]**************************************************
every batch cost: 0.4680147171020508
**************************************************
every batch cost: 0.47008824348449707
Epoch 1/50:  90%|████████▉ | 35/39 [00:56<00:02,  1.88it/s, lr=0.001, step/s=0.613, total_loss=445]**************************************************
every batch cost: 0.6110391616821289
**************************************************
every batch cost: 0.5674312114715576
Epoch 1/50:  95%|█████████▍| 37/39 [00:57<00:01,  1.76it/s, lr=0.001, step/s=0.612, total_loss=425]**************************************************
every batch cost: 0.6096856594085693
**************************************************
every batch cost: 0.5798914432525635
Epoch 1/50: 100%|██████████| 39/39 [00:58<00:00,  1.69it/s, lr=0.001, step/s=0.623, total_loss=406]**************************************************
every batch cost: 0.621854305267334
Epoch 1/50: 100%|██████████| 39/39 [00:58<00:00,  1.51s/it, lr=0.001, step/s=0.623, total_loss=406]
Epoch 1/50:   0%|          | 0/4 [00:00<?, ?it/s<class 'dict'>]Start Validation
Epoch 1/50: 100%|██████████| 4/4 [00:09<00:00,  2.36s/it, total_loss=53.2]
Finish Validation
Epoch:1/50
Total Loss: 396.0543 || Val Loss: 42.5870 
Saving state, iter: 1
Epoch 2/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 39.29310464859009
**************************************************
every batch cost: 0.7208483219146729
Epoch 2/50:   5%|▌         | 2/39 [00:42<17:40, 28.66s/it, lr=0.000905, step/s=2.17, total_loss=55.5]**************************************************
every batch cost: 0.728407621383667
Epoch 2/50:   8%|▊         | 3/39 [00:43<12:14, 20.39s/it, lr=0.000905, step/s=1.08, total_loss=54.6]**************************************************
every batch cost: 1.0782456398010254
Epoch 2/50:  10%|█         | 4/39 [00:44<08:30, 14.58s/it, lr=0.000905, step/s=1.01, total_loss=53.8]**************************************************
every batch cost: 1.004713773727417
**************************************************
every batch cost: 0.9956800937652588
Epoch 2/50:  15%|█▌        | 6/39 [00:45<04:08,  7.54s/it, lr=0.000905, step/s=0.622, total_loss=52]**************************************************
every batch cost: 0.6200478076934814
Epoch 2/50:  18%|█▊        | 7/39 [00:46<02:53,  5.42s/it, lr=0.000905, step/s=0.458, total_loss=51.3]**************************************************
every batch cost: 0.46021366119384766
**************************************************
every batch cost: 0.5946834087371826
Epoch 2/50:  23%|██▎       | 9/39 [00:47<01:28,  2.96s/it, lr=0.000905, step/s=0.565, total_loss=49.9]**************************************************
every batch cost: 0.5631444454193115
**************************************************
every batch cost: 0.44608592987060547
Epoch 2/50:  28%|██▊       | 11/39 [00:48<00:48,  1.73s/it, lr=0.000905, step/s=0.609, total_loss=48.7]**************************************************
every batch cost: 0.608043909072876
Epoch 2/50:  31%|███       | 12/39 [00:49<00:36,  1.36s/it, lr=0.000905, step/s=0.486, total_loss=48.1]**************************************************
every batch cost: 0.4828364849090576
Epoch 2/50:  33%|███▎      | 13/39 [00:49<00:28,  1.09s/it, lr=0.000905, step/s=0.465, total_loss=47.6]**************************************************
every batch cost: 0.4629390239715576
**************************************************
every batch cost: 0.5994398593902588
Epoch 2/50:  38%|███▊      | 15/39 [00:50<00:20,  1.18it/s, lr=0.000905, step/s=0.606, total_loss=46.6]**************************************************
every batch cost: 0.6047122478485107
Epoch 2/50:  41%|████      | 16/39 [00:51<00:17,  1.32it/s, lr=0.000905, step/s=0.536, total_loss=46]**************************************************
every batch cost: 0.5342566967010498
**************************************************
every batch cost: 0.5950450897216797
Epoch 2/50:  46%|████▌     | 18/39 [00:52<00:14,  1.45it/s, lr=0.000905, step/s=0.617, total_loss=45]**************************************************
every batch cost: 0.6158950328826904
**************************************************
every batch cost: 0.46120595932006836
Epoch 2/50:  51%|█████▏    | 20/39 [00:53<00:10,  1.74it/s, lr=0.000905, step/s=0.454, total_loss=44.1]**************************************************
every batch cost: 0.4514193534851074
**************************************************
every batch cost: 0.566706657409668
Epoch 2/50:  56%|█████▋    | 22/39 [00:54<00:09,  1.82it/s, lr=0.000905, step/s=0.48, total_loss=43.2]**************************************************
every batch cost: 0.47788500785827637
**************************************************
every batch cost: 0.4445042610168457
Epoch 2/50:  62%|██████▏   | 24/39 [00:55<00:08,  1.82it/s, lr=0.000905, step/s=0.608, total_loss=42.4]**************************************************
every batch cost: 0.6049983501434326
Epoch 2/50:  64%|██████▍   | 25/39 [00:56<00:07,  1.91it/s, lr=0.000905, step/s=0.456, total_loss=42]**************************************************
every batch cost: 0.4528319835662842
Epoch 2/50:  67%|██████▋   | 26/39 [00:56<00:06,  2.01it/s, lr=0.000905, step/s=0.421, total_loss=41.6]**************************************************
every batch cost: 0.4186275005340576
Epoch 2/50:  69%|██████▉   | 27/39 [00:57<00:05,  2.10it/s, lr=0.000905, step/s=0.416, total_loss=41.2]**************************************************
every batch cost: 0.4143967628479004
**************************************************
every batch cost: 0.6277880668640137
Epoch 2/50:  74%|███████▍  | 29/39 [00:58<00:04,  2.07it/s, lr=0.000905, step/s=0.381, total_loss=40.5]**************************************************
every batch cost: 0.37793684005737305
**************************************************
every batch cost: 0.5735199451446533
Epoch 2/50:  79%|███████▉  | 31/39 [00:59<00:04,  1.82it/s, lr=0.000905, step/s=0.627, total_loss=39.8]**************************************************
every batch cost: 0.6239285469055176
Epoch 2/50:  82%|████████▏ | 32/39 [00:59<00:03,  1.91it/s, lr=0.000905, step/s=0.447, total_loss=39.4]**************************************************
every batch cost: 0.4438314437866211
Epoch 2/50:  85%|████████▍ | 33/39 [01:00<00:03,  1.98it/s, lr=0.000905, step/s=0.455, total_loss=39.1]**************************************************
every batch cost: 0.45426464080810547
**************************************************
every batch cost: 0.4659461975097656
Epoch 2/50:  90%|████████▉ | 35/39 [01:01<00:02,  1.90it/s, lr=0.000905, step/s=0.588, total_loss=38.5]**************************************************
every batch cost: 0.5880696773529053
**************************************************
every batch cost: 0.560051441192627
Epoch 2/50:  95%|█████████▍| 37/39 [01:02<00:01,  1.93it/s, lr=0.000905, step/s=0.459, total_loss=37.9]**************************************************
every batch cost: 0.4551582336425781
Epoch 2/50:  97%|█████████▋| 38/39 [01:02<00:00,  1.98it/s, lr=0.000905, step/s=0.464, total_loss=37.6]**************************************************
every batch cost: 0.46265411376953125
Epoch 2/50: 100%|██████████| 39/39 [01:03<00:00,  1.85it/s, lr=0.000905, step/s=0.619, total_loss=37.3]**************************************************
every batch cost: 0.6277220249176025
Epoch 2/50: 100%|██████████| 39/39 [01:03<00:00,  1.63s/it, lr=0.000905, step/s=0.619, total_loss=37.3]
Epoch 2/50:   0%|          | 0/4 [00:00<?, ?it/s<class 'dict'>]Start Validation
Epoch 2/50: 100%|██████████| 4/4 [00:09<00:00,  2.41s/it, total_loss=24.6]
Finish Validation
Epoch:2/50
Total Loss: 36.4114 || Val Loss: 19.6662 
Saving state, iter: 2
Epoch 3/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 35.0421621799469
**************************************************
every batch cost: 1.201897382736206
Epoch 3/50:   5%|▌         | 2/39 [00:43<16:55, 27.45s/it, lr=0.000658, step/s=6.93, total_loss=26.2]**************************************************
every batch cost: 1.0342633724212646
Epoch 3/50:   8%|▊         | 3/39 [00:44<11:43, 19.55s/it, lr=0.000658, step/s=1.11, total_loss=26]**************************************************
every batch cost: 1.1082875728607178
Epoch 3/50:  10%|█         | 4/39 [00:45<08:12, 14.07s/it, lr=0.000658, step/s=1.25, total_loss=25.9]**************************************************
every batch cost: 1.2519311904907227
Epoch 3/50:  13%|█▎        | 5/39 [00:46<05:45, 10.16s/it, lr=0.000658, step/s=1.03, total_loss=25.6]**************************************************
every batch cost: 1.0359077453613281
**************************************************
every batch cost: 0.8948409557342529
Epoch 3/50:  18%|█▊        | 7/39 [00:47<02:49,  5.30s/it, lr=0.000658, step/s=0.439, total_loss=25.4]**************************************************
every batch cost: 0.4398157596588135
Epoch 3/50:  21%|██        | 8/39 [00:48<02:01,  3.91s/it, lr=0.000658, step/s=0.647, total_loss=25.2]**************************************************
every batch cost: 0.6475496292114258
**************************************************
every batch cost: 0.441436767578125
Epoch 3/50:  26%|██▌       | 10/39 [00:49<01:02,  2.15s/it, lr=0.000658, step/s=0.45, total_loss=25]**************************************************
every batch cost: 0.4504427909851074
**************************************************
every batch cost: 0.659254789352417
Epoch 3/50:  31%|███       | 12/39 [00:50<00:35,  1.33s/it, lr=0.000658, step/s=0.448, total_loss=24.8]**************************************************
every batch cost: 0.4467794895172119
Epoch 3/50:  33%|███▎      | 13/39 [00:51<00:27,  1.07s/it, lr=0.000658, step/s=0.451, total_loss=24.6]**************************************************
every batch cost: 0.45061635971069336
Epoch 3/50:  36%|███▌      | 14/39 [00:51<00:22,  1.12it/s, lr=0.000658, step/s=0.463, total_loss=24.5]**************************************************
every batch cost: 0.46184635162353516
**************************************************
every batch cost: 0.45751118659973145
Epoch 3/50:  41%|████      | 16/39 [00:52<00:15,  1.47it/s, lr=0.000658, step/s=0.484, total_loss=24.3]**************************************************
every batch cost: 0.4834902286529541
Epoch 3/50:  44%|████▎     | 17/39 [00:53<00:13,  1.59it/s, lr=0.000658, step/s=0.493, total_loss=24.2]**************************************************
every batch cost: 0.49303245544433594
**************************************************
every batch cost: 0.4651303291320801
Epoch 3/50:  49%|████▊     | 19/39 [00:54<00:11,  1.72it/s, lr=0.000658, step/s=0.571, total_loss=24]**************************************************
every batch cost: 0.5711841583251953
**************************************************
every batch cost: 0.6247999668121338
Epoch 3/50:  54%|█████▍    | 21/39 [00:55<00:10,  1.67it/s, lr=0.000658, step/s=0.595, total_loss=23.8]**************************************************
every batch cost: 0.5951683521270752
**************************************************
every batch cost: 0.4792053699493408
Epoch 3/50:  59%|█████▉    | 23/39 [00:56<00:08,  1.84it/s, lr=0.000658, step/s=0.482, total_loss=23.6]**************************************************
every batch cost: 0.4827558994293213
Epoch 3/50:  62%|██████▏   | 24/39 [00:56<00:07,  1.93it/s, lr=0.000658, step/s=0.446, total_loss=23.5]**************************************************
every batch cost: 0.4450535774230957
**************************************************
every batch cost: 0.4633326530456543
Epoch 3/50:  67%|██████▋   | 26/39 [00:57<00:06,  2.05it/s, lr=0.000658, step/s=0.439, total_loss=23.2]**************************************************
every batch cost: 0.43566155433654785
**************************************************
every batch cost: 0.6175928115844727
Epoch 3/50:  72%|███████▏  | 28/39 [00:58<00:05,  1.99it/s, lr=0.000658, step/s=0.431, total_loss=23]**************************************************
every batch cost: 0.42892026901245117
Epoch 3/50:  74%|███████▍  | 29/39 [00:59<00:05,  1.88it/s, lr=0.000658, step/s=0.594, total_loss=23]**************************************************
every batch cost: 0.5934576988220215
**************************************************
every batch cost: 0.4818403720855713
Epoch 3/50:  79%|███████▉  | 31/39 [01:00<00:03,  2.01it/s, lr=0.000658, step/s=0.438, total_loss=22.8]**************************************************
every batch cost: 0.4389064311981201
**************************************************
every batch cost: 0.5992274284362793
Epoch 3/50:  85%|████████▍ | 33/39 [01:01<00:03,  1.88it/s, lr=0.000658, step/s=0.522, total_loss=22.7]**************************************************
every batch cost: 0.5205366611480713
**************************************************
every batch cost: 0.6633074283599854
Epoch 3/50:  90%|████████▉ | 35/39 [01:02<00:02,  1.71it/s, lr=0.000658, step/s=0.608, total_loss=22.5]**************************************************
every batch cost: 0.6174895763397217
Epoch 3/50:  92%|█████████▏| 36/39 [01:03<00:01,  1.88it/s, lr=0.000658, step/s=0.395, total_loss=22.4]**************************************************
every batch cost: 0.39395880699157715
Epoch 3/50:  95%|█████████▍| 37/39 [01:03<00:01,  1.94it/s, lr=0.000658, step/s=0.465, total_loss=22.3]**************************************************
every batch cost: 0.46547436714172363
Epoch 3/50:  97%|█████████▋| 38/39 [01:04<00:00,  1.81it/s, lr=0.000658, step/s=0.628, total_loss=22.2]**************************************************
every batch cost: 0.6294529438018799
Epoch 3/50: 100%|██████████| 39/39 [01:04<00:00,  1.92it/s, lr=0.000658, step/s=0.436, total_loss=22.1]**************************************************
every batch cost: 0.43511104583740234
Epoch 3/50: 100%|██████████| 39/39 [01:05<00:00,  1.67s/it, lr=0.000658, step/s=0.436, total_loss=22.1]
Epoch 3/50:   0%|          | 0/4 [00:00<?, ?it/s<class 'dict'>]Start Validation
Epoch 3/50: 100%|██████████| 4/4 [00:09<00:00,  2.42s/it, total_loss=17]
Finish Validation
Epoch:3/50
Total Loss: 21.5840 || Val Loss: 13.6119 
Saving state, iter: 3
Epoch 4/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 36.35298943519592
Epoch 4/50:   3%|▎         | 1/39 [00:37<23:48, 37.58s/it, lr=0.000352, step/s=37.6, total_loss=19.2]**************************************************
every batch cost: 1.2303707599639893
**************************************************
every batch cost: 1.144068956375122
Epoch 4/50:   8%|▊         | 3/39 [00:39<11:22, 18.96s/it, lr=0.000352, step/s=0.999, total_loss=18.6]**************************************************
every batch cost: 0.9969470500946045
**************************************************
Epoch 4/50:  10%|█         | 4/39 [00:40<07:56, 13.61s/it, lr=0.000352, step/s=0.878, total_loss=18.5]every batch cost: 1.114208698272705
**************************************************
every batch cost: 0.7834813594818115
Epoch 4/50:  15%|█▌        | 6/39 [00:43<03:58,  7.23s/it, lr=0.000352, step/s=1.17, total_loss=18.3]**************************************************
every batch cost: 1.1689801216125488
**************************************************
every batch cost: 1.0269944667816162
Epoch 4/50:  21%|██        | 8/39 [00:45<02:05,  4.06s/it, lr=0.000352, step/s=0.997, total_loss=18.3]**************************************************
every batch cost: 0.8472988605499268
**************************************************
every batch cost: 1.1264011859893799
Epoch 4/50:  26%|██▌       | 10/39 [00:47<01:14,  2.58s/it, lr=0.000352, step/s=1.03, total_loss=18.3]**************************************************
every batch cost: 1.317326545715332
**************************************************
every batch cost: 0.6522595882415771
Epoch 4/50:  31%|███       | 12/39 [00:48<00:44,  1.64s/it, lr=0.000352, step/s=0.676, total_loss=18.3]**************************************************
every batch cost: 0.6746888160705566
Epoch 4/50:  33%|███▎      | 13/39 [00:49<00:35,  1.35s/it, lr=0.000352, step/s=0.644, total_loss=18.3]**************************************************
every batch cost: 0.6441326141357422
Epoch 4/50:  36%|███▌      | 14/39 [00:50<00:27,  1.09s/it, lr=0.000352, step/s=0.466, total_loss=18.3]**************************************************
every batch cost: 0.4652385711669922
Epoch 4/50:  38%|███▊      | 15/39 [00:50<00:21,  1.12it/s, lr=0.000352, step/s=0.444, total_loss=18.2]**************************************************
every batch cost: 0.4434363842010498
**************************************************
every batch cost: 0.6525671482086182
Epoch 4/50:  44%|████▎     | 17/39 [00:51<00:15,  1.40it/s, lr=0.000352, step/s=0.448, total_loss=18.2]**************************************************
every batch cost: 0.44656848907470703
**************************************************
every batch cost: 0.46964049339294434
Epoch 4/50:  49%|████▊     | 19/39 [00:52<00:12,  1.56it/s, lr=0.000352, step/s=0.623, total_loss=18.1]**************************************************
every batch cost: 0.6219379901885986
**************************************************
every batch cost: 0.5789623260498047
Epoch 4/50:  54%|█████▍    | 21/39 [00:53<00:10,  1.71it/s, lr=0.000352, step/s=0.482, total_loss=18]**************************************************
every batch cost: 0.48101806640625
Epoch 4/50:  56%|█████▋    | 22/39 [00:54<00:10,  1.68it/s, lr=0.000352, step/s=0.605, total_loss=18]**************************************************
every batch cost: 0.6064701080322266
**************************************************
every batch cost: 0.5328128337860107
Epoch 4/50:  62%|██████▏   | 24/39 [00:55<00:08,  1.82it/s, lr=0.000352, step/s=0.468, total_loss=17.9]**************************************************
every batch cost: 0.46857547760009766
Epoch 4/50:  64%|██████▍   | 25/39 [00:56<00:07,  1.77it/s, lr=0.000352, step/s=0.597, total_loss=17.8]**************************************************
every batch cost: 0.59743332862854
Epoch 4/50:  67%|██████▋   | 26/39 [00:56<00:07,  1.72it/s, lr=0.000352, step/s=0.611, total_loss=17.8]**************************************************
every batch cost: 0.6083073616027832
Epoch 4/50:  69%|██████▉   | 27/39 [00:57<00:07,  1.66it/s, lr=0.000352, step/s=0.641, total_loss=17.8]**************************************************
every batch cost: 0.6388955116271973
**************************************************
every batch cost: 0.6160726547241211
Epoch 4/50:  74%|███████▍  | 29/39 [00:58<00:05,  1.80it/s, lr=0.000352, step/s=0.418, total_loss=17.7]**************************************************
every batch cost: 0.4156637191772461
Epoch 4/50:  77%|███████▋  | 30/39 [00:58<00:04,  1.87it/s, lr=0.000352, step/s=0.479, total_loss=17.7]**************************************************
every batch cost: 0.4801647663116455
**************************************************
every batch cost: 0.6667578220367432
Epoch 4/50:  82%|████████▏ | 32/39 [01:00<00:04,  1.67it/s, lr=0.000352, step/s=0.638, total_loss=17.6]**************************************************
every batch cost: 0.6348106861114502
Epoch 4/50:  85%|████████▍ | 33/39 [01:00<00:03,  1.63it/s, lr=0.000352, step/s=0.632, total_loss=17.6]**************************************************
every batch cost: 0.6294233798980713
Epoch 4/50:  87%|████████▋ | 34/39 [01:01<00:03,  1.63it/s, lr=0.000352, step/s=0.601, total_loss=17.5]**************************************************
every batch cost: 0.599083662033081
Epoch 4/50:  90%|████████▉ | 35/39 [01:02<00:02,  1.62it/s, lr=0.000352, step/s=0.619, total_loss=17.5]**************************************************
every batch cost: 0.6171004772186279
Epoch 4/50:  92%|█████████▏| 36/39 [01:02<00:01,  1.61it/s, lr=0.000352, step/s=0.625, total_loss=17.5]**************************************************
every batch cost: 0.6238992214202881
Epoch 4/50:  95%|█████████▍| 37/39 [01:03<00:01,  1.56it/s, lr=0.000352, step/s=0.676, total_loss=17.5]**************************************************
every batch cost: 0.6731662750244141
Epoch 4/50:  97%|█████████▋| 38/39 [01:04<00:00,  1.54it/s, lr=0.000352, step/s=0.659, total_loss=17.5]**************************************************
every batch cost: 0.6571424007415771
Epoch 4/50: 100%|██████████| 39/39 [01:04<00:00,  1.57it/s, lr=0.000352, step/s=0.599, total_loss=17.4]**************************************************
every batch cost: 0.5965077877044678
Epoch 4/50: 100%|██████████| 39/39 [01:05<00:00,  1.67s/it, lr=0.000352, step/s=0.599, total_loss=17.4]
Epoch 4/50:   0%|          | 0/4 [00:00<?, ?it/s<class 'dict'>]Start Validation
Epoch 4/50: 100%|██████████| 4/4 [00:09<00:00,  2.45s/it, total_loss=14.4]
Finish Validation
Epoch:4/50
Total Loss: 17.0058 || Val Loss: 11.5347 
Saving state, iter: 4
Epoch 5/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 38.64342498779297
Epoch 5/50:   3%|▎         | 1/39 [00:39<24:55, 39.35s/it, lr=0.000105, step/s=39.3, total_loss=16]**************************************************
every batch cost: 0.709841251373291
Epoch 5/50:   5%|▌         | 2/39 [00:40<17:10, 27.84s/it, lr=0.000105, step/s=0.973, total_loss=15.5]**************************************************
every batch cost: 0.972780704498291





num_workers=16, pin_memory=True
############################################
Epoch 1/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 14.460282325744629
Epoch 1/50:   3%|▎         | 1/39 [00:16<10:45, 16.98s/it, lr=0.001, step/s=17, total_loss=1.86e+3]**************************************************
every batch cost: 2.522810220718384
Epoch 1/50:   5%|▌         | 2/39 [00:28<09:32, 15.47s/it, lr=0.001, step/s=11.9, total_loss=1.75e+3]**************************************************
every batch cost: 0.7985043525695801
**************************************************
every batch cost: 1.024996042251587
Epoch 1/50:  10%|█         | 4/39 [00:31<04:49,  8.28s/it, lr=0.001, step/s=1.62, total_loss=1.56e+3]**************************************************
every batch cost: 1.6076982021331787
Epoch 1/50:  13%|█▎        | 5/39 [00:32<03:29,  6.17s/it, lr=0.001, step/s=1.23, total_loss=1.48e+3]**************************************************
every batch cost: 1.2147128582000732
Epoch 1/50:  15%|█▌        | 6/39 [00:34<02:35,  4.72s/it, lr=0.001, step/s=1.33, total_loss=1.4e+3]**************************************************
every batch cost: 1.3201313018798828
**************************************************
every batch cost: 1.310088872909546
Epoch 1/50:  21%|██        | 8/39 [00:36<01:31,  2.96s/it, lr=0.001, step/s=1.23, total_loss=1.26e+3]**************************************************
every batch cost: 1.2326617240905762
**************************************************
every batch cost: 1.2288479804992676
Epoch 1/50:  26%|██▌       | 10/39 [00:39<01:01,  2.12s/it, lr=0.001, step/s=1.33, total_loss=1.14e+3]**************************************************
every batch cost: 1.3325657844543457
Epoch 1/50:  28%|██▊       | 11/39 [00:40<00:51,  1.85s/it, lr=0.001, step/s=1.21, total_loss=1.08e+3]**************************************************
every batch cost: 1.2089898586273193
Epoch 1/50:  31%|███       | 12/39 [00:42<00:46,  1.74s/it, lr=0.001, step/s=1.32, total_loss=1.03e+3]**************************************************
every batch cost: 1.170703649520874
Epoch 1/50:  33%|███▎      | 13/39 [00:42<00:35,  1.36s/it, lr=0.001, step/s=0.475, total_loss=983]**************************************************
every batch cost: 0.4666435718536377
Epoch 1/50:  36%|███▌      | 14/39 [00:43<00:29,  1.18s/it, lr=0.001, step/s=0.733, total_loss=940]**************************************************
every batch cost: 0.7289862632751465
**************************************************
every batch cost: 0.454329252243042
Epoch 1/50:  41%|████      | 16/39 [00:44<00:18,  1.23it/s, lr=0.001, step/s=0.449, total_loss=862]**************************************************
every batch cost: 0.44701433181762695
**************************************************
every batch cost: 0.46754884719848633
Epoch 1/50:  46%|████▌     | 18/39 [00:45<00:13,  1.60it/s, lr=0.001, step/s=0.415, total_loss=795]**************************************************
every batch cost: 0.41454458236694336
**************************************************
every batch cost: 0.4329829216003418
Epoch 1/50:  51%|█████▏    | 20/39 [00:45<00:10,  1.84it/s, lr=0.001, step/s=0.47, total_loss=737]**************************************************
every batch cost: 0.46698427200317383
**************************************************
every batch cost: 0.44852232933044434
Epoch 1/50:  56%|█████▋    | 22/39 [00:47<00:09,  1.85it/s, lr=0.001, step/s=0.59, total_loss=686]**************************************************
every batch cost: 0.5892877578735352
**************************************************
every batch cost: 0.5070881843566895
Epoch 1/50:  62%|██████▏   | 24/39 [00:48<00:08,  1.83it/s, lr=0.001, step/s=0.565, total_loss=642]**************************************************
every batch cost: 0.5660033226013184
**************************************************
every batch cost: 0.4133646488189697
Epoch 1/50:  67%|██████▋   | 26/39 [00:49<00:06,  2.02it/s, lr=0.001, step/s=0.451, total_loss=602]**************************************************
every batch cost: 0.4506559371948242
**************************************************
every batch cost: 0.625328779220581
Epoch 1/50:  72%|███████▏  | 28/39 [00:50<00:05,  1.93it/s, lr=0.001, step/s=0.471, total_loss=567]**************************************************
every batch cost: 0.46766138076782227
**************************************************
every batch cost: 0.4441709518432617
Epoch 1/50:  77%|███████▋  | 30/39 [00:51<00:04,  2.07it/s, lr=0.001, step/s=0.439, total_loss=536]**************************************************
every batch cost: 0.4360997676849365
**************************************************
every batch cost: 0.4722580909729004
Epoch 1/50:  82%|████████▏ | 32/39 [00:51<00:03,  2.11it/s, lr=0.001, step/s=0.439, total_loss=508]**************************************************
every batch cost: 0.43885302543640137
Epoch 1/50:  85%|████████▍ | 33/39 [00:52<00:02,  2.08it/s, lr=0.001, step/s=0.486, total_loss=495]**************************************************
every batch cost: 0.485029935836792
Epoch 1/50:  87%|████████▋ | 34/39 [00:52<00:02,  2.18it/s, lr=0.001, step/s=0.402, total_loss=483]**************************************************
every batch cost: 0.39949965476989746
Epoch 1/50:  90%|████████▉ | 35/39 [00:53<00:01,  2.15it/s, lr=0.001, step/s=0.473, total_loss=471]**************************************************
every batch cost: 0.4719250202178955
Epoch 1/50:  92%|█████████▏| 36/39 [00:53<00:01,  2.21it/s, lr=0.001, step/s=0.415, total_loss=460]**************************************************
every batch cost: 0.4150104522705078
Epoch 1/50:  95%|█████████▍| 37/39 [00:54<00:00,  2.14it/s, lr=0.001, step/s=0.493, total_loss=449]**************************************************
every batch cost: 0.4902806282043457
Epoch 1/50:  97%|█████████▋| 38/39 [00:54<00:00,  2.16it/s, lr=0.001, step/s=0.439, total_loss=439]**************************************************
every batch cost: 0.4379861354827881
Epoch 1/50: 100%|██████████| 39/39 [00:55<00:00,  2.15it/s, lr=0.001, step/s=0.466, total_loss=430]**************************************************
every batch cost: 0.46781086921691895
Start Validation
Epoch 1/50: 100%|██████████| 39/39 [00:55<00:00,  1.42s/it, lr=0.001, step/s=0.466, total_loss=430]
Epoch 1/50: 100%|██████████| 4/4 [00:08<00:00,  2.07s/it, total_loss=58.2]
Finish Validation
Epoch:1/50
Total Loss: 418.9609 || Val Loss: 46.5974 
Saving state, iter: 1
Epoch 2/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 15.21496319770813
**************************************************
every batch cost: 0.6695339679718018
Epoch 2/50:   5%|▌         | 2/39 [00:24<08:31, 13.83s/it, lr=0.000905, step/s=9.04, total_loss=59]**************************************************
every batch cost: 0.5902526378631592
Epoch 2/50:   8%|▊         | 3/39 [00:26<06:02, 10.07s/it, lr=0.000905, step/s=1.28, total_loss=57.9]**************************************************
every batch cost: 1.288323163986206
Epoch 2/50:  10%|█         | 4/39 [00:31<05:04,  8.71s/it, lr=0.000905, step/s=5.53, total_loss=57.2]**************************************************
every batch cost: 0.9619312286376953
Epoch 2/50:  13%|█▎        | 5/39 [00:33<03:41,  6.50s/it, lr=0.000905, step/s=1.34, total_loss=56.1]**************************************************
every batch cost: 1.33282470703125

num_workers=8, pin_memory=True
########################################################

**************************************************
0.0002262592315673828
Epoch 1/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 14.393999099731445
Epoch 1/50:   3%|▎         | 1/39 [00:16<10:28, 16.53s/it, lr=0.001, step/s=16.5, total_loss=1.66e+3]**************************************************
every batch cost: 2.138085126876831
Epoch 1/50:   5%|▌         | 2/39 [00:17<07:13, 11.73s/it, lr=0.001, step/s=0.51, total_loss=1.56e+3]**************************************************
every batch cost: 0.4993324279785156
Epoch 1/50:   8%|▊         | 3/39 [00:17<05:02,  8.39s/it, lr=0.001, step/s=0.6, total_loss=1.47e+3]**************************************************
every batch cost: 0.5929081439971924
Epoch 1/50:  10%|█         | 4/39 [00:18<03:31,  6.03s/it, lr=0.001, step/s=0.525, total_loss=1.39e+3]**************************************************
every batch cost: 0.515371561050415
Epoch 1/50:  13%|█▎        | 5/39 [00:18<02:28,  4.37s/it, lr=0.001, step/s=0.5, total_loss=1.31e+3]**************************************************
every batch cost: 0.4886586666107178
**************************************************
every batch cost: 0.7947738170623779
Epoch 1/50:  18%|█▊        | 7/39 [00:20<01:23,  2.61s/it, lr=0.001, step/s=0.991, total_loss=1.17e+3]**************************************************
every batch cost: 0.9867894649505615
Epoch 1/50:  21%|██        | 8/39 [00:21<01:09,  2.24s/it, lr=0.001, step/s=1.37, total_loss=1.11e+3]**************************************************
every batch cost: 1.365293025970459
Epoch 1/50:  23%|██▎       | 9/39 [00:25<01:17,  2.60s/it, lr=0.001, step/s=3.42, total_loss=1.05e+3]**************************************************
every batch cost: 0.9516646862030029
**************************************************
every batch cost: 0.5585477352142334
Epoch 1/50:  28%|██▊       | 11/39 [00:26<00:44,  1.57s/it, lr=0.001, step/s=0.592, total_loss=950]**************************************************
every batch cost: 0.5857737064361572
**************************************************
every batch cost: 0.48770785331726074
Epoch 1/50:  33%|███▎      | 13/39 [00:27<00:26,  1.04s/it, lr=0.001, step/s=0.531, total_loss=863]**************************************************
every batch cost: 0.5245354175567627
Epoch 1/50:  36%|███▌      | 14/39 [00:28<00:22,  1.10it/s, lr=0.001, step/s=0.603, total_loss=825]**************************************************
every batch cost: 0.6007485389709473
Epoch 1/50:  38%|███▊      | 15/39 [00:28<00:19,  1.26it/s, lr=0.001, step/s=0.522, total_loss=789]**************************************************
every batch cost: 0.5292763710021973
**************************************************
every batch cost: 1.456395149230957
Epoch 1/50:  44%|████▎     | 17/39 [00:34<00:41,  1.87s/it, lr=0.001, step/s=3.91, total_loss=726]**************************************************
every batch cost: 0.8605103492736816
**************************************************
every batch cost: 0.9400577545166016
Epoch 1/50:  49%|████▊     | 19/39 [00:35<00:25,  1.28s/it, lr=0.001, step/s=0.539, total_loss=670]**************************************************
every batch cost: 0.5312719345092773
**************************************************
every batch cost: 0.6191096305847168
Epoch 1/50:  54%|█████▍    | 21/39 [00:36<00:16,  1.06it/s, lr=0.001, step/s=0.587, total_loss=623]**************************************************
every batch cost: 0.5655641555786133
Epoch 1/50:  56%|█████▋    | 22/39 [00:37<00:13,  1.26it/s, lr=0.001, step/s=0.451, total_loss=601]**************************************************
every batch cost: 0.44422078132629395
Epoch 1/50:  59%|█████▉    | 23/39 [00:38<00:13,  1.15it/s, lr=0.001, step/s=1.03, total_loss=581]**************************************************
every batch cost: 1.0400919914245605
**************************************************
every batch cost: 1.245389461517334
Epoch 1/50:  64%|██████▍   | 25/39 [00:42<00:21,  1.52s/it, lr=0.001, step/s=2.77, total_loss=544]**************************************************
every batch cost: 0.874835729598999
Epoch 1/50:  67%|██████▋   | 26/39 [00:43<00:17,  1.34s/it, lr=0.001, step/s=0.912, total_loss=527]**************************************************
every batch cost: 0.9050905704498291
**************************************************
every batch cost: 0.6378293037414551
Epoch 1/50:  72%|███████▏  | 28/39 [00:44<00:10,  1.04it/s, lr=0.001, step/s=0.556, total_loss=497]**************************************************
every batch cost: 0.5520014762878418
Epoch 1/50:  74%|███████▍  | 29/39 [00:44<00:08,  1.21it/s, lr=0.001, step/s=0.507, total_loss=483]**************************************************
every batch cost: 0.5015432834625244
**************************************************
every batch cost: 0.5492942333221436
Epoch 1/50:  79%|███████▉  | 31/39 [00:46<00:05,  1.45it/s, lr=0.001, step/s=0.546, total_loss=457]**************************************************
every batch cost: 0.5536196231842041
Epoch 1/50:  82%|████████▏ | 32/39 [00:46<00:04,  1.55it/s, lr=0.001, step/s=0.534, total_loss=445]**************************************************
every batch cost: 0.5281355381011963
Epoch 1/50:  85%|████████▍ | 33/39 [00:50<00:09,  1.65s/it, lr=0.001, step/s=3.97, total_loss=434]**************************************************
every batch cost: 1.134850025177002
**************************************************
every batch cost: 0.41797971725463867
Epoch 1/50:  90%|████████▉ | 35/39 [00:51<00:04,  1.05s/it, lr=0.001, step/s=0.497, total_loss=413]**************************************************
every batch cost: 0.4928009510040283
Epoch 1/50:  92%|█████████▏| 36/39 [00:51<00:02,  1.15it/s, lr=0.001, step/s=0.429, total_loss=403]**************************************************
every batch cost: 0.4249401092529297
**************************************************
every batch cost: 0.44547200202941895
Epoch 1/50:  97%|█████████▋| 38/39 [00:52<00:00,  1.54it/s, lr=0.001, step/s=0.418, total_loss=385]**************************************************
every batch cost: 0.4139523506164551
**************************************************
every batch cost: 0.49946165084838867
Epoch 1/50: 100%|██████████| 39/39 [00:53<00:00,  1.37s/it, lr=0.001, step/s=0.491, total_loss=377]
Start Validation
Epoch 1/50: 100%|██████████| 4/4 [00:07<00:00,  1.87s/it, total_loss=51.2]
Finish Validation
Epoch:1/50
Total Loss: 367.2327 || Val Loss: 40.9737 
Saving state, iter: 1
Epoch 2/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 15.813380241394043
Epoch 2/50:   3%|▎         | 1/39 [00:16<10:43, 16.92s/it, lr=0.000905, step/s=16.9, total_loss=53.5]**************************************************
every batch cost: 1.1080009937286377
**************************************************
every batch cost: 0.898298978805542
Epoch 2/50:   8%|▊         | 3/39 [00:18<05:10,  8.62s/it, lr=0.000905, step/s=0.466, total_loss=51.2]**************************************************
every batch cost: 0.4567604064941406
**************************************************
every batch cost: 0.5346977710723877
Epoch 2/50:  13%|█▎        | 5/39 [00:19<02:32,  4.49s/it, lr=0.000905, step/s=0.502, total_loss=50.1]**************************************************
every batch cost: 0.494734525680542
**************************************************
every batch cost: 0.5638992786407471
Epoch 2/50:  18%|█▊        | 7/39 [00:20<01:22,  2.57s/it, lr=0.000905, step/s=0.818, total_loss=48.6]**************************************************
every batch cost: 0.812410831451416
**************************************************
every batch cost: 0.8895196914672852
Epoch 2/50:  23%|██▎       | 9/39 [00:22<00:55,  1.83s/it, lr=0.000905, step/s=1.28, total_loss=47.1]**************************************************
every batch cost: 0.803992748260498
Saving state, iter: 2
Epoch 3/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 13.237895488739014

num_workers=4, pin_memory=True
########################################################
**************************************************
0.00022721290588378906
Epoch 1/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 8.96416425704956
**************************************************
every batch cost: 2.4007461071014404
Epoch 1/50:   5%|▌         | 2/39 [00:11<04:59,  8.10s/it, lr=0.001, step/s=0.486, total_loss=1.55e+3]**************************************************
every batch cost: 0.4803886413574219
Epoch 1/50:   8%|▊         | 3/39 [00:12<03:29,  5.82s/it, lr=0.001, step/s=0.496, total_loss=1.46e+3]**************************************************
every batch cost: 0.5041465759277344
Epoch 1/50:  10%|█         | 4/39 [00:12<02:27,  4.22s/it, lr=0.001, step/s=0.458, total_loss=1.37e+3]**************************************************
every batch cost: 0.45255351066589355
**************************************************
every batch cost: 0.48969101905822754
Epoch 1/50:  15%|█▌        | 6/39 [00:19<01:55,  3.51s/it, lr=0.001, step/s=0.438, total_loss=1.23e+3]**************************************************
every batch cost: 0.4459989070892334
Epoch 1/50:  18%|█▊        | 7/39 [00:19<01:23,  2.60s/it, lr=0.001, step/s=0.472, total_loss=1.16e+3]**************************************************
every batch cost: 0.4651622772216797
**************************************************
every batch cost: 0.48007702827453613
Epoch 1/50:  23%|██▎       | 9/39 [00:26<01:38,  3.30s/it, lr=0.001, step/s=6.4, total_loss=1.04e+3]**************************************************
every batch cost: 0.437375545501709
**************************************************
every batch cost: 0.4270005226135254
Epoch 1/50:  28%|██▊       | 11/39 [00:27<00:52,  1.87s/it, lr=0.001, step/s=0.534, total_loss=943]**************************************************
every batch cost: 0.5334141254425049
**************************************************
every batch cost: 0.432300329208374
Epoch 1/50:  33%|███▎      | 13/39 [00:34<01:13,  2.81s/it, lr=0.001, step/s=6, total_loss=857]**************************************************
every batch cost: 0.40425610542297363
Epoch 1/50:  36%|███▌      | 14/39 [00:34<00:52,  2.11s/it, lr=0.001, step/s=0.47, total_loss=819]**************************************************
every batch cost: 0.479872465133667
Epoch 1/50:  38%|███▊      | 15/39 [00:35<00:38,  1.61s/it, lr=0.001, step/s=0.418, total_loss=783]**************************************************
every batch cost: 0.4144725799560547
**************************************************
every batch cost: 0.5075886249542236
Epoch 1/50:  44%|████▎     | 17/39 [00:42<01:01,  2.81s/it, lr=0.001, step/s=6.37, total_loss=720]**************************************************
every batch cost: 0.8164219856262207
**************************************************
every batch cost: 0.4030587673187256
Epoch 1/50:  49%|████▊     | 19/39 [00:43<00:32,  1.62s/it, lr=0.001, step/s=0.492, total_loss=665]**************************************************
every batch cost: 0.4989743232727051
Epoch 1/50:  51%|█████▏    | 20/39 [00:43<00:24,  1.30s/it, lr=0.001, step/s=0.554, total_loss=641]**************************************************
every batch cost: 0.5493001937866211
**************************************************
every batch cost: 0.8048973083496094
Epoch 1/50:  56%|█████▋    | 22/39 [00:50<00:34,  2.06s/it, lr=0.001, step/s=0.461, total_loss=596]**************************************************
every batch cost: 0.4469301700592041
Epoch 1/50:  59%|█████▉    | 23/39 [00:50<00:25,  1.58s/it, lr=0.001, step/s=0.455, total_loss=576]**************************************************
every batch cost: 0.46288108825683594
Epoch 1/50:  62%|██████▏   | 24/39 [00:51<00:19,  1.31s/it, lr=0.001, step/s=0.689, total_loss=558]**************************************************
every batch cost: 0.6864700317382812
Epoch 1/50:  64%|██████▍   | 25/39 [00:57<00:37,  2.70s/it, lr=0.001, step/s=5.92, total_loss=540]**************************************************
every batch cost: 0.7928135395050049
Epoch 1/50:  67%|██████▋   | 26/39 [00:57<00:26,  2.02s/it, lr=0.001, step/s=0.445, total_loss=523]**************************************************
every batch cost: 0.43711113929748535
**************************************************
every batch cost: 0.4797182083129883
Epoch 1/50:  72%|███████▏  | 28/39 [00:58<00:13,  1.22s/it, lr=0.001, step/s=0.408, total_loss=493]**************************************************
every batch cost: 0.40384721755981445
**************************************************
every batch cost: 0.7942979335784912
Epoch 1/50:  77%|███████▋  | 30/39 [01:05<00:18,  2.09s/it, lr=0.001, step/s=0.441, total_loss=466]**************************************************
every batch cost: 0.4379851818084717
Epoch 1/50:  79%|███████▉  | 31/39 [01:06<00:12,  1.62s/it, lr=0.001, step/s=0.504, total_loss=453]**************************************************
every batch cost: 0.5120210647583008
Epoch 1/50:  82%|████████▏ | 32/39 [01:06<00:08,  1.28s/it, lr=0.001, step/s=0.484, total_loss=442]**************************************************
every batch cost: 0.4807112216949463
Epoch 1/50:  85%|████████▍ | 33/39 [01:12<00:16,  2.81s/it, lr=0.001, step/s=6.38, total_loss=430]**************************************************
every batch cost: 0.9237005710601807
Epoch 1/50:  87%|████████▋ | 34/39 [01:13<00:10,  2.14s/it, lr=0.001, step/s=0.548, total_loss=420]**************************************************
every batch cost: 0.546823263168335
Epoch 1/50:  90%|████████▉ | 35/39 [01:13<00:06,  1.63s/it, lr=0.001, step/s=0.446, total_loss=410]**************************************************
every batch cost: 0.4558219909667969
**************************************************
every batch cost: 0.5195400714874268
Epoch 1/50:  95%|█████████▍| 37/39 [01:19<00:05,  2.51s/it, lr=0.001, step/s=5.32, total_loss=391]**************************************************
every batch cost: 0.7924432754516602
**************************************************
every batch cost: 0.4358391761779785
Epoch 1/50: 100%|██████████| 39/39 [01:20<00:00,  1.45s/it, lr=0.001, step/s=0.394, total_loss=374]**************************************************
every batch cost: 0.403045654296875
Start Validation
Epoch 1/50: 100%|██████████| 39/39 [01:21<00:00,  2.08s/it, lr=0.001, step/s=0.394, total_loss=374]
Epoch 1/50: 100%|██████████| 4/4 [00:07<00:00,  1.89s/it, total_loss=49.5]
Finish Validation
Epoch:1/50
Total Loss: 364.2570 || Val Loss: 39.5904 
Saving state, iter: 1
Epoch 2/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
8.968660593032837
Epoch 2/50:   3%|▎         | 1/39 [00:09<06:12,  9.82s/it, lr=0.000905, step/s=9.81, total_loss=52.4]**************************************************
every batch cost: 0.8473787307739258
Epoch 2/50:   5%|▌         | 2/39 [00:12<04:39,  7.55s/it, lr=0.000905, step/s=2.24, total_loss=51.9]**************************************************
every batch cost: 0.46636486053466797
**************************************************
every batch cost: 0.4795670509338379
Epoch 2/50:  10%|█         | 4/39 [00:13<02:17,  3.94s/it, lr=0.000905, step/s=0.464, total_loss=50.6]**************************************************
every batch cost: 0.46152782440185547

Epoch 3/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 10.565217733383179

num_workers=2, pin_memory=True
########################################################
**************************************************
0.0007519721984863281
Epoch 1/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 8.411113977432251
**************************************************
every batch cost: 1.3904120922088623
Epoch 1/50:   5%|▌         | 2/39 [00:10<04:19,  7.02s/it, lr=0.001, step/s=0.51, total_loss=1.66e+3]**************************************************
every batch cost: 0.5176153182983398
Epoch 1/50:   8%|▊         | 3/39 [00:15<03:56,  6.57s/it, lr=0.001, step/s=5.53, total_loss=1.56e+3]**************************************************
every batch cost: 0.441662073135376
**************************************************
every batch cost: 0.4310026168823242
Epoch 1/50:  13%|█▎        | 5/39 [00:22<02:58,  5.25s/it, lr=0.001, step/s=6.47, total_loss=1.39e+3]**************************************************
every batch cost: 0.4786345958709717
**************************************************
every batch cost: 0.5005531311035156
Epoch 1/50:  18%|█▊        | 7/39 [00:29<02:29,  4.66s/it, lr=0.001, step/s=6.59, total_loss=1.25e+3]**************************************************
every batch cost: 0.5762255191802979
Epoch 1/50:  21%|██        | 8/39 [00:30<01:45,  3.41s/it, lr=0.001, step/s=0.506, total_loss=1.18e+3]**************************************************
every batch cost: 0.5124270915985107
Epoch 1/50:  23%|██▎       | 9/39 [00:36<02:09,  4.33s/it, lr=0.001, step/s=6.45, total_loss=1.12e+3]**************************************************
every batch cost: 0.5291411876678467
Epoch 1/50:  26%|██▌       | 10/39 [00:37<01:32,  3.19s/it, lr=0.001, step/s=0.537, total_loss=1.07e+3]**************************************************
every batch cost: 0.5446579456329346
**************************************************
every batch cost: 0.40877699851989746
Epoch 1/50:  31%|███       | 12/39 [00:44<01:20,  3.00s/it, lr=0.001, step/s=0.379, total_loss=968]**************************************************
every batch cost: 0.38829779624938965
**************************************************
every batch cost: 0.4400508403778076
Epoch 1/50:  36%|███▌      | 14/39 [00:51<01:14,  2.98s/it, lr=0.001, step/s=0.459, total_loss=883]**************************************************
every batch cost: 0.4682137966156006
**************************************************
every batch cost: 0.4381704330444336
Epoch 1/50:  41%|████      | 16/39 [00:57<01:07,  2.96s/it, lr=0.001, step/s=0.4, total_loss=810]**************************************************
every batch cost: 0.4084131717681885
**************************************************
every batch cost: 0.44188928604125977
Epoch 1/50:  46%|████▌     | 18/39 [01:05<01:02,  2.99s/it, lr=0.001, step/s=0.476, total_loss=747]**************************************************
every batch cost: 0.48462367057800293
Epoch 1/50:  49%|████▊     | 19/39 [01:11<01:21,  4.06s/it, lr=0.001, step/s=6.54, total_loss=719]**************************************************
every batch cost: 0.4357719421386719
Epoch 1/50:  51%|█████▏    | 20/39 [01:12<00:56,  2.96s/it, lr=0.001, step/s=0.41, total_loss=692]**************************************************
every batch cost: 0.41820716857910156
**************************************************
every batch cost: 0.3943486213684082
Epoch 1/50:  56%|█████▋    | 22/39 [01:19<00:50,  2.97s/it, lr=0.001, step/s=0.431, total_loss=645]**************************************************
every batch cost: 0.43925046920776367
**************************************************
every batch cost: 0.4390239715576172
Epoch 1/50:  62%|██████▏   | 24/39 [01:25<00:44,  2.94s/it, lr=0.001, step/s=0.382, total_loss=603]**************************************************
every batch cost: 0.39272522926330566
Epoch 1/50:  64%|██████▍   | 25/39 [01:32<00:56,  4.01s/it, lr=0.001, step/s=6.5, total_loss=584]**************************************************
every batch cost: 0.4005303382873535
Epoch 1/50:  67%|██████▋   | 26/39 [01:32<00:38,  2.93s/it, lr=0.001, step/s=0.413, total_loss=566]**************************************************
every batch cost: 0.4216008186340332
**************************************************
every batch cost: 0.3795938491821289
Epoch 1/50:  72%|███████▏  | 28/39 [01:40<00:34,  3.10s/it, lr=0.001, step/s=0.52, total_loss=533]**************************************************
every batch cost: 0.5263004302978516
Epoch 1/50:  74%|███████▍  | 29/39 [01:47<00:41,  4.11s/it, lr=0.001, step/s=6.44, total_loss=518]**************************************************
every batch cost: 0.3753941059112549
**************************************************
every batch cost: 0.3916609287261963
Epoch 1/50:  79%|███████▉  | 31/39 [01:54<00:34,  4.30s/it, lr=0.001, step/s=7.34, total_loss=490]**************************************************
every batch cost: 0.4356727600097656
**************************************************
every batch cost: 0.42064833641052246
Epoch 1/50:  85%|████████▍ | 33/39 [02:02<00:25,  4.23s/it, lr=0.001, step/s=6.79, total_loss=465]**************************************************
every batch cost: 0.44506239891052246
Epoch 1/50:  87%|████████▋ | 34/39 [02:02<00:15,  3.10s/it, lr=0.001, step/s=0.427, total_loss=454]**************************************************
every batch cost: 0.44188857078552246
**************************************************
every batch cost: 0.4287247657775879
Epoch 1/50:  92%|█████████▏| 36/39 [02:09<00:09,  3.11s/it, lr=0.001, step/s=0.408, total_loss=432]**************************************************
every batch cost: 0.41641998291015625
Epoch 1/50:  95%|█████████▍| 37/39 [02:16<00:08,  4.04s/it, lr=0.001, step/s=6.2, total_loss=422]**************************************************
every batch cost: 0.4182100296020508
Epoch 1/50:  97%|█████████▋| 38/39 [02:16<00:02,  2.96s/it, lr=0.001, step/s=0.414, total_loss=413]**************************************************
every batch cost: 0.42250919342041016
Epoch 1/50: 100%|██████████| 39/39 [02:22<00:00,  3.95s/it, lr=0.001, step/s=6.25, total_loss=404]**************************************************
every batch cost: 0.3862333297729492
Epoch 1/50: 100%|██████████| 39/39 [02:22<00:00,  3.67s/it, lr=0.001, step/s=6.25, total_loss=404]
Epoch 1/50:   0%|          | 0/4 [00:00<?, ?it/s<class 'dict'>]Start Validation
Epoch 1/50: 100%|██████████| 4/4 [00:09<00:00,  2.34s/it, total_loss=55]
Finish Validation
Epoch:1/50
Total Loss: 393.7135 || Val Loss: 43.9618 
Saving state, iter: 1
Epoch 2/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 9.354033470153809
Epoch 2/50:   3%|▎         | 1/39 [00:09<06:12,  9.81s/it, lr=0.000905, step/s=9.81, total_loss=57.9]**************************************************
every batch cost: 0.46102428436279297
Epoch 2/50:   5%|▌         | 2/39 [00:10<04:19,  7.01s/it, lr=0.000905, step/s=0.443, total_loss=56.8]**************************************************
every batch cost: 0.44997072219848633

Total Loss: 37.1114 || Val Loss: 19.7967 
Saving state, iter: 2
Epoch 3/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 8.828293561935425
Saving state, iter: 3
Epoch 4/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
8.770357131958008
Epoch 4/50:   3%|▎   





num_workers=1, pin_memory=True
###############################################
Epoch 1/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 7.561102628707886
Epoch 1/50:   3%|▎         | 1/39 [00:08<05:39,  8.93s/it, lr=0.001, step/s=8.93, total_loss=1.47e+3]**************************************************
every batch cost: 1.3696913719177246
**************************************************
every batch cost: 0.4643228054046631
Epoch 1/50:   8%|▊         | 3/39 [00:21<04:34,  7.62s/it, lr=0.001, step/s=6.52, total_loss=1.3e+3]**************************************************
every batch cost: 0.41152310371398926
Epoch 1/50:  10%|█         | 4/39 [00:28<04:17,  7.36s/it, lr=0.001, step/s=6.74, total_loss=1.22e+3]**************************************************
every batch cost: 0.4380664825439453
Epoch 1/50:  13%|█▎        | 5/39 [00:34<04:02,  7.13s/it, lr=0.001, step/s=6.57, total_loss=1.15e+3]**************************************************
every batch cost: 0.39557337760925293
Epoch 1/50:  15%|█▌        | 6/39 [00:41<03:51,  7.01s/it, lr=0.001, step/s=6.72, total_loss=1.09e+3]**************************************************
every batch cost: 0.43558669090270996
Epoch 1/50:  18%|█▊        | 7/39 [00:48<03:41,  6.93s/it, lr=0.001, step/s=6.73, total_loss=1.03e+3]**************************************************
every batch cost: 0.47762060165405273
Epoch 1/50:  21%|██        | 8/39 [00:54<03:30,  6.79s/it, lr=0.001, step/s=6.47, total_loss=972]**************************************************
every batch cost: 0.450481653213501
**************************************************
every batch cost: 0.6359219551086426
Epoch 1/50:  26%|██▌       | 10/39 [01:07<03:10,  6.58s/it, lr=0.001, step/s=6.19, total_loss=875]**************************************************
every batch cost: 0.3963806629180908
**************************************************
every batch cost: 0.40274739265441895
Epoch 1/50:  31%|███       | 12/39 [01:21<03:00,  6.67s/it, lr=0.001, step/s=6.77, total_loss=792]**************************************************
every batch cost: 0.43964362144470215
Epoch 1/50:  33%|███▎      | 13/39 [01:27<02:54,  6.71s/it, lr=0.001, step/s=6.79, total_loss=755]**************************************************
every batch cost: 0.42827844619750977
**************************************************
every batch cost: 0.502422571182251
Epoch 1/50:  38%|███▊      | 15/39 [01:41<02:40,  6.67s/it, lr=0.001, step/s=6.73, total_loss=690]**************************************************
every batch cost: 0.5271012783050537
**************************************************
every batch cost: 0.5445053577423096
Epoch 1/50:  44%|████▎     | 17/39 [01:54<02:28,  6.73s/it, lr=0.001, step/s=6.84, total_loss=634]**************************************************
every batch cost: 0.5539577007293701
Epoch 1/50:  46%|████▌     | 18/39 [02:01<02:22,  6.80s/it, lr=0.001, step/s=6.95, total_loss=609]**************************************************
every batch cost: 0.5347979068756104
Epoch 1/50:  49%|████▊     | 19/39 [02:08<02:13,  6.69s/it, lr=0.001, step/s=6.42, total_loss=586]**************************************************
every batch cost: 0.5063800811767578
**************************************************
every batch cost: 0.5031697750091553
Epoch 1/50:  54%|█████▍    | 21/39 [02:21<02:00,  6.70s/it, lr=0.001, step/s=6.78, total_loss=544]**************************************************
every batch cost: 0.6011106967926025
Epoch 1/50:  56%|█████▋    | 22/39 [02:28<01:53,  6.70s/it, lr=0.001, step/s=6.69, total_loss=525]**************************************************
every batch cost: 0.5622284412384033
**************************************************
every batch cost: 0.5850467681884766
Epoch 1/50:  62%|██████▏   | 24/39 [02:41<01:41,  6.75s/it, lr=0.001, step/s=6.76, total_loss=491]**************************************************
every batch cost: 0.498795747756958
Epoch 1/50:  64%|██████▍   | 25/39 [02:48<01:34,  6.78s/it, lr=0.001, step/s=6.86, total_loss=475]**************************************************
every batch cost: 0.5354490280151367
Epoch 1/50:  67%|██████▋   | 26/39 [02:55<01:28,  6.82s/it, lr=0.001, step/s=6.89, total_loss=460]**************************************************
every batch cost: 0.5751655101776123
Epoch 1/50:  69%|██████▉   | 27/39 [03:02<01:23,  6.92s/it, lr=0.001, step/s=7.16, total_loss=447]**************************************************
every batch cost: 0.6159842014312744
Epoch 1/50:  72%|███████▏  | 28/39 [03:09<01:14,  6.74s/it, lr=0.001, step/s=6.31, total_loss=434]**************************************************
every batch cost: 0.38555479049682617
**************************************************
every batch cost: 0.42298007011413574
Epoch 1/50:  77%|███████▋  | 30/39 [03:22<01:00,  6.77s/it, lr=0.001, step/s=6.72, total_loss=410]**************************************************
every batch cost: 0.3904249668121338
Epoch 1/50:  79%|███████▉  | 31/39 [03:29<00:54,  6.76s/it, lr=0.001, step/s=6.73, total_loss=399]**************************************************
every batch cost: 0.44513821601867676
Epoch 1/50:  82%|████████▏ | 32/39 [03:36<00:47,  6.78s/it, lr=0.001, step/s=6.82, total_loss=389]**************************************************
every batch cost: 0.3929564952850342
**************************************************
every batch cost: 0.4448988437652588
Epoch 1/50:  87%|████████▋ | 34/39 [03:49<00:33,  6.72s/it, lr=0.001, step/s=6.61, total_loss=369]**************************************************
every batch cost: 0.40523266792297363
**************************************************
every batch cost: 0.3851611614227295
Epoch 1/50:  92%|█████████▏| 36/39 [04:03<00:20,  6.76s/it, lr=0.001, step/s=6.79, total_loss=352]**************************************************
every batch cost: 0.43050265312194824
**************************************************
every batch cost: 0.41400766372680664
Epoch 1/50:  97%|█████████▋| 38/39 [04:16<00:06,  6.67s/it, lr=0.001, step/s=6.65, total_loss=336]**************************************************
every batch cost: 0.4372570514678955
Epoch 1/50: 100%|██████████| 39/39 [04:22<00:00,  6.60s/it, lr=0.001, step/s=6.43, total_loss=329]**************************************************
every batch cost: 0.37398743629455566
Epoch 1/50: 100%|██████████| 39/39 [04:22<00:00,  6.74s/it, lr=0.001, step/s=6.43, total_loss=329]
Epoch 1/50:   0%|          | 0/4 [00:00<?, ?it/s<class 'dict'>]Start Validation
Epoch 1/50: 100%|██████████| 4/4 [00:14<00:00,  3.59s/it, total_loss=43.1]
Finish Validation
Epoch:1/50
Total Loss: 320.5249 || Val Loss: 34.4960 
Saving state, iter: 1
Epoch 2/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 8.003284931182861
**************************************************
every batch cost: 0.5475771427154541
Epoch 2/50:   5%|▌         | 2/39 [00:15<04:59,  8.10s/it, lr=0.000905, step/s=7.03, total_loss=46.8]**************************************************
every batch cost: 0.5343887805938721
Epoch 2/50:   8%|▊         | 3/39 [00:22<04:37,  7.70s/it, lr=0.000905, step/s=6.78, total_loss=45.9]**************************************************
every batch cost: 0.5383634567260742
Epoch 2/50:  10%|█         | 4/39 [00:29<04:19,  7.40s/it, lr=0.000905, step/s=6.7, total_loss=45.1]**************************************************
every batch cost: 0.5001688003540039
**************************************************
every batch cost: 0.571533203125
Epoch 2/50:  15%|█▌        | 6/39 [00:42<03:55,  7.14s/it, lr=0.000905, step/s=6.8, total_loss=43.6]**************************************************
every batch cost: 0.49525880813598633
Epoch 2/50:  18%|█▊        | 7/39 [00:49<03:44,  7.02s/it, lr=0.000905, step/s=6.73, total_loss=43]**************************************************
every batch cost: 0.40282368659973145
**************************************************
every batch cost: 0.40811753273010254
Epoch 2/50:  23%|██▎       | 9/39 [01:03<03:30,  7.01s/it, lr=0.000905, step/s=7.09, total_loss=41.8]**************************************************
every batch cost: 0.4412691593170166
**************************************************
every batch cost: 0.37858128547668457
Epoch 2/50:  28%|██▊       | 11/39 [01:17<03:14,  6.96s/it, lr=0.000905, step/s=6.9, total_loss=40.7]**************************************************
every batch cost: 0.42623257637023926
Epoch 2/50:  31%|███       | 12/39 [01:24<03:09,  7.03s/it, lr=0.000905, step/s=7.19, total_loss=40.3]**************************************************
every batch cost: 0.45459723472595215
**************************************************
every batch cost: 0.49076294898986816
Epoch 2/50:  36%|███▌      | 14/39 [01:38<02:51,  6.87s/it, lr=0.000905, step/s=6.7, total_loss=39.3]**************************************************
every batch cost: 0.570073127746582
**************************************************
every batch cost: 0.4507887363433838
Epoch 2/50:  41%|████      | 16/39 [01:51<02:36,  6.80s/it, lr=0.000905, step/s=6.73, total_loss=38.4]**************************************************
every batch cost: 0.41584324836730957
Epoch 2/50:  44%|████▎     | 17/39 [01:58<02:29,  6.80s/it, lr=0.000905, step/s=6.77, total_loss=37.9]**************************************************
every batch cost: 0.3880279064178467
Epoch 2/50:  46%|████▌     | 18/39 [02:04<02:22,  6.76s/it, lr=0.000905, step/s=6.68, total_loss=37.6]**************************************************
every batch cost: 0.4323999881744385
Epoch 2/50:  49%|████▊     | 19/39 [02:11<02:15,  6.78s/it, lr=0.000905, step/s=6.82, total_loss=37.2]**************************************************
every batch cost: 0.4374668598175049
**************************************************
every batch cost: 0.38514113426208496
Epoch 2/50:  54%|█████▍    | 21/39 [02:25<02:02,  6.81s/it, lr=0.000905, step/s=6.71, total_loss=36.5]**************************************************
every batch cost: 0.46169018745422363
**************************************************
every batch cost: 0.4422013759613037
Epoch 2/50:  59%|█████▉    | 23/39 [02:39<01:50,  6.88s/it, lr=0.000905, step/s=6.9, total_loss=35.8]**************************************************
every batch cost: 0.42540407180786133
**************************************************
every batch cost: 0.44600915908813477
Epoch 2/50:  64%|██████▍   | 25/39 [02:53<01:35,  6.85s/it, lr=0.000905, step/s=6.9, total_loss=35.2]**************************************************
every batch cost: 0.3933718204498291
**************************************************
every batch cost: 0.4039616584777832
Epoch 2/50:  69%|██████▉   | 27/39 [03:06<01:21,  6.77s/it, lr=0.000905, step/s=6.72, total_loss=34.6]**************************************************
every batch cost: 0.40546417236328125
**************************************************
every batch cost: 0.3811759948730469
Epoch 2/50:  74%|███████▍  | 29/39 [03:19<01:07,  6.75s/it, lr=0.000905, step/s=6.8, total_loss=33.9]**************************************************
every batch cost: 0.3452949523925781
**************************************************
every batch cost: 0.39229869842529297
Epoch 2/50:  79%|███████▉  | 31/39 [03:33<00:54,  6.81s/it, lr=0.000905, step/s=6.76, total_loss=33.4]**************************************************
every batch cost: 0.3768949508666992
Epoch 2/50:  82%|████████▏ | 32/39 [03:40<00:47,  6.84s/it, lr=0.000905, step/s=6.89, total_loss=33.1]**************************************************
every batch cost: 0.39880943298339844
**************************************************
every batch cost: 0.45096421241760254
Epoch 2/50:  87%|████████▋ | 34/39 [03:54<00:34,  6.84s/it, lr=0.000905, step/s=6.78, total_loss=32.6]**************************************************
every batch cost: 0.4104328155517578
Epoch 2/50:  90%|████████▉ | 35/39 [04:01<00:27,  6.84s/it, lr=0.000905, step/s=6.83, total_loss=32.3]**************************************************
every batch cost: 0.4044673442840576
Epoch 2/50:  92%|█████████▏| 36/39 [04:07<00:20,  6.75s/it, lr=0.000905, step/s=6.54, total_loss=32]**************************************************
every batch cost: 0.3817863464355469
Epoch 2/50:  95%|█████████▍| 37/39 [04:14<00:13,  6.70s/it, lr=0.000905, step/s=6.58, total_loss=31.8]**************************************************
every batch cost: 0.3974142074584961
Epoch 2/50:  97%|█████████▋| 38/39 [04:21<00:06,  6.73s/it, lr=0.000905, step/s=6.79, total_loss=31.6]**************************************************
every batch cost: 0.3678750991821289
**************************************************
every batch cost: 0.40433621406555176
Epoch 2/50: 100%|██████████| 39/39 [04:28<00:00,  6.87s/it, lr=0.000905, step/s=6.91, total_loss=31.3]
Start Validation
Epoch 2/50: 100%|██████████| 4/4 [00:13<00:00,  3.50s/it, total_loss=20.4]
Finish Validation
Epoch:2/50
Total Loss: 30.5174 || Val Loss: 16.3263 
Saving state, iter: 2
Epoch 3/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 7.984367370605469
**************************************************
every batch cost: 0.63826584815979
Epoch 3/50:   5%|▌         | 2/39 [00:15<04:56,  8.01s/it, lr=0.000658, step/s=6.56, total_loss=22]**************************************************
every batch cost: 0.4317927360534668
Epoch 3/50:   8%|▊         | 3/39 [00:22<04:36,  7.67s/it, lr=0.000658, step/s=6.88, total_loss=21.8]**************************************************
every batch cost: 0.4045250415802002
Epoch 3/50:  10%|█         | 4/39 [00:29<04:23,  7.53s/it, lr=0.000658, step/s=7.17, total_loss=21.9]**************************************************
every batch cost: 0.5850446224212646
**************************************************
every batch cost: 0.45392608642578125
Epoch 3/50:  15%|█▌        | 6/39 [00:42<03:52,  7.05s/it, lr=0.000658, step/s=6.54, total_loss=21.7]**************************************************
every batch cost: 0.42398643493652344
Epoch 3/50:  18%|█▊        | 7/39 [00:49<03:44,  7.00s/it, lr=0.000658, step/s=6.9, total_loss=21.5]**************************************************
every batch cost: 0.5161738395690918
**************************************************
every batch cost: 0.4052703380584717
Epoch 3/50:  23%|██▎       | 9/39 [01:02<03:24,  6.82s/it, lr=0.000658, step/s=6.72, total_loss=21.3]**************************************************
every batch cost: 0.40622735023498535
**************************************************
every batch cost: 0.4583563804626465
Epoch 3/50:  28%|██▊       | 11/39 [01:16<03:11,  6.84s/it, lr=0.000658, step/s=6.75, total_loss=21.3]**************************************************
every batch cost: 0.45088720321655273
**************************************************
every batch cost: 0.41858887672424316
Epoch 3/50:  33%|███▎      | 13/39 [01:29<02:57,  6.81s/it, lr=0.000658, step/s=6.73, total_loss=20.9]**************************************************
every batch cost: 0.5316376686096191
Epoch 3/50:  36%|███▌      | 14/39 [01:36<02:50,  6.81s/it, lr=0.000658, step/s=6.8, total_loss=20.8]**************************************************
every batch cost: 0.43830227851867676
Epoch 3/50:  38%|███▊      | 15/39 [01:43<02:45,  6.92s/it, lr=0.000658, step/s=7.15, total_loss=20.7]**************************************************
every batch cost: 0.5127363204956055
**************************************************
every batch cost: 0.5016570091247559
Epoch 3/50:  44%|████▎     | 17/39 [01:57<02:31,  6.88s/it, lr=0.000658, step/s=6.71, total_loss=20.6]**************************************************
every batch cost: 0.5417201519012451
**************************************************
every batch cost: 0.48558616638183594
Epoch 3/50:  49%|████▊     | 19/39 [02:11<02:18,  6.94s/it, lr=0.000658, step/s=7.08, total_loss=20.4]**************************************************
every batch cost: 0.5570189952850342
**************************************************
every batch cost: 0.5443787574768066
Epoch 3/50:  54%|█████▍    | 21/39 [02:24<02:02,  6.80s/it, lr=0.000658, step/s=6.59, total_loss=20.2]**************************************************
every batch cost: 0.5134713649749756
**************************************************
every batch cost: 0.5383806228637695
Epoch 3/50:  59%|█████▉    | 23/39 [02:38<01:50,  6.89s/it, lr=0.000658, step/s=6.94, total_loss=20.2]**************************************************
every batch cost: 0.600865364074707
Epoch 3/50:  62%|██████▏   | 24/39 [02:46<01:44,  6.94s/it, lr=0.000658, step/s=7.03, total_loss=20.1]**************************************************
every batch cost: 0.5180280208587646
Epoch 3/50:  64%|██████▍   | 25/39 [02:52<01:36,  6.90s/it, lr=0.000658, step/s=6.82, total_loss=20]**************************************************
every batch cost: 0.5188186168670654
Epoch 3/50:  67%|██████▋   | 26/39 [02:59<01:29,  6.90s/it, lr=0.000658, step/s=6.88, total_loss=19.9]**************************************************
every batch cost: 0.446591854095459
**************************************************
every batch cost: 0.614121675491333
Epoch 3/50:  72%|███████▏  | 28/39 [03:13<01:16,  6.96s/it, lr=0.000658, step/s=6.91, total_loss=19.8]**************************************************
every batch cost: 0.5028769969940186
Epoch:3/50
Total Loss: 18.5234 || Val Loss: 11.3431 
Saving state, iter: 3
Epoch 4/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 8.067720651626587
Epoch 4/50:   3%|▎         | 1/39 [00:08<05:23,  8.51s/it, lr=0.000352, step/s=8.51, total_loss=16.7]**************************************************
every batch cost: 0.4458169937133789






num_workers=0, pin_memory=True,
###############################################

Loading weights into state dict...
Finished!
**************************************************
0.00026345252990722656
Epoch 1/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 6.736436128616333
**************************************************
every batch cost: 1.3526716232299805
Epoch 1/50:   5%|▌         | 2/39 [00:14<04:44,  7.70s/it, lr=0.001, step/s=6.78, total_loss=1.64e+3]**************************************************
every batch cost: 0.4258887767791748
**************************************************
every batch cost: 0.5254902839660645
Epoch 1/50:  10%|█         | 4/39 [00:28<04:12,  7.21s/it, lr=0.001, step/s=6.68, total_loss=1.46e+3]**************************************************
every batch cost: 0.4723479747772217
Epoch 1/50:  13%|█▎        | 5/39 [00:34<03:57,  6.98s/it, lr=0.001, step/s=6.42, total_loss=1.38e+3]**************************************************
every batch cost: 0.3637819290161133
**************************************************
every batch cost: 0.39740514755249023
Epoch 1/50:  18%|█▊        | 7/39 [00:48<03:37,  6.81s/it, lr=0.001, step/s=6.63, total_loss=1.23e+3]**************************************************
every batch cost: 0.3908064365386963
**************************************************
every batch cost: 0.4139695167541504
Epoch 1/50:  23%|██▎       | 9/39 [01:01<03:24,  6.83s/it, lr=0.001, step/s=6.78, total_loss=1.11e+3]**************************************************
every batch cost: 0.3817429542541504
Epoch 1/50:  26%|██▌       | 10/39 [01:08<03:15,  6.75s/it, lr=0.001, step/s=6.58, total_loss=1.05e+3]**************************************************
every batch cost: 0.35744786262512207
**************************************************
every batch cost: 0.45519328117370605
Epoch 1/50:  31%|███       | 12/39 [01:22<03:03,  6.78s/it, lr=0.001, step/s=6.75, total_loss=955]**************************************************
every batch cost: 0.389538049697876
Epoch 1/50:  33%|███▎      | 13/39 [01:28<02:56,  6.77s/it, lr=0.001, step/s=6.74, total_loss=912]**************************************************
every batch cost: 0.3983135223388672
**************************************************
every batch cost: 0.4398193359375
Epoch 1/50:  38%|███▊      | 15/39 [01:42<02:42,  6.79s/it, lr=0.001, step/s=6.76, total_loss=834]**************************************************
every batch cost: 0.42572808265686035
**************************************************
every batch cost: 0.4335670471191406
Epoch 1/50:  44%|████▎     | 17/39 [01:56<02:29,  6.81s/it, lr=0.001, step/s=6.8, total_loss=767]**************************************************
every batch cost: 0.4214038848876953
Epoch 1/50:  46%|████▌     | 18/39 [02:02<02:21,  6.76s/it, lr=0.001, step/s=6.63, total_loss=737]**************************************************
every batch cost: 0.3599708080291748
**************************************************
every batch cost: 0.4076557159423828
Epoch 1/50:  51%|█████▏    | 20/39 [02:16<02:08,  6.78s/it, lr=0.001, step/s=6.9, total_loss=683]**************************************************
every batch cost: 0.432664155960083
Epoch 1/50:  54%|█████▍    | 21/39 [02:22<02:01,  6.75s/it, lr=0.001, step/s=6.66, total_loss=658]**************************************************
every batch cost: 0.4187169075012207
**************************************************
every batch cost: 0.4376835823059082
Epoch 1/50:  59%|█████▉    | 23/39 [02:36<01:48,  6.80s/it, lr=0.001, step/s=6.8, total_loss=614]**************************************************
every batch cost: 0.41230297088623047
Epoch 1/50:  62%|██████▏   | 24/39 [02:43<01:41,  6.78s/it, lr=0.001, step/s=6.71, total_loss=594]**************************************************
every batch cost: 0.42777347564697266
**************************************************
every batch cost: 0.416337251663208
Epoch 1/50:  67%|██████▋   | 26/39 [02:57<01:29,  6.85s/it, lr=0.001, step/s=6.91, total_loss=558]**************************************************
every batch cost: 0.43065834045410156
Epoch 1/50:  69%|██████▉   | 27/39 [03:04<01:22,  6.86s/it, lr=0.001, step/s=6.87, total_loss=541]**************************************************
every batch cost: 0.4243924617767334
Epoch 1/50:  72%|███████▏  | 28/39 [03:10<01:15,  6.84s/it, lr=0.001, step/s=6.79, total_loss=525]**************************************************
every batch cost: 0.4390888214111328
Epoch 1/50:  74%|███████▍  | 29/39 [03:17<01:08,  6.87s/it, lr=0.001, step/s=6.94, total_loss=510]**************************************************
every batch cost: 0.44049930572509766
**************************************************
every batch cost: 0.4194369316101074
Epoch 1/50:  79%|███████▉  | 31/39 [03:31<00:54,  6.85s/it, lr=0.001, step/s=6.79, total_loss=483]**************************************************
every batch cost: 0.416440486907959
Epoch 1/50:  82%|████████▏ | 32/39 [03:38<00:47,  6.85s/it, lr=0.001, step/s=6.83, total_loss=470]**************************************************
every batch cost: 0.45289063453674316
**************************************************
every batch cost: 0.43844008445739746
Epoch 1/50:  87%|████████▋ | 34/39 [03:52<00:34,  6.88s/it, lr=0.001, step/s=6.78, total_loss=447]**************************************************
every batch cost: 0.40239405632019043
Epoch 1/50:  90%|████████▉ | 35/39 [03:59<00:27,  6.87s/it, lr=0.001, step/s=6.82, total_loss=436]**************************************************
every batch cost: 0.3687868118286133
**************************************************
every batch cost: 0.43868470191955566
Epoch 1/50:  95%|█████████▍| 37/39 [04:12<00:13,  6.82s/it, lr=0.001, step/s=6.65, total_loss=416]**************************************************
every batch cost: 0.39241647720336914
**************************************************
every batch cost: 0.4395029544830322
Epoch 1/50: 100%|██████████| 39/39 [04:26<00:00,  6.83s/it, lr=0.001, step/s=6.69, total_loss=398]
Epoch 1/50:   0%|          | 0/4 [00:00<?, ?it/s<class 'dict'>]**************************************************
every batch cost: 0.4003739356994629
Start Validation
Epoch 1/50: 100%|██████████| 4/4 [00:12<00:00,  3.01s/it, total_loss=53.4]
Finish Validation
Epoch:1/50
Total Loss: 387.7016 || Val Loss: 42.7095 
Saving state, iter: 1
Epoch 2/50:   0%|          | 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************
load data cost: 6.612175464630127
**************************************************
every batch cost: 0.4096841812133789
Epoch 2/50:   5%|▌         | 2/39 [00:13<04:17,  6.97s/it, lr=0.000905, step/s=6.84, total_loss=53.3]**************************************************
every batch cost: 0.4010329246520996
Epoch 2/50:   8%|▊         | 3/39 [00:20<04:09,  6.92s/it, lr=0.000905, step/s=6.79, total_loss=52.8]**************************************************
every batch cost: 0.40485644340515137
**************************************************
every batch cost: 0.3858015537261963