-
Notifications
You must be signed in to change notification settings - Fork 0
/
nohup.out
2303 lines (2283 loc) · 323 KB
/
nohup.out
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2022-09-07 11:49:09 [INFO]
------------Environment Information-------------
platform: Linux-4.15.0-140-generic-x86_64-with-debian-stretch-sid
Python: 3.7.4 (default, Aug 13 2019, 20:35:49) [GCC 7.3.0]
Paddle compiled with cuda: True
NVCC: Build cuda_11.2.r11.2/compiler.29618528_0
cudnn: 8.2
GPUs used: 1
CUDA_VISIBLE_DEVICES: None
GPU: ['GPU 0: Tesla V100-SXM2-32GB']
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~16.04) 7.5.0
PaddlePaddle: 2.3.2
------------------------------------------------
Traceback (most recent call last):
File "train.py", line 204, in <module>
main(args)
File "train.py", line 149, in main
batch_size=args.batch_size)
File "/home/aistudio/MedicalSeg/medicalseg/cvlibs/config.py", line 80, in __init__
raise FileNotFoundError('File {} does not exist'.format(path))
FileNotFoundError: File MedicalSeg/configs/trans_unet/trans_unet_synapse.yml does not exist
2022-09-07 11:49:50 [INFO]
------------Environment Information-------------
platform: Linux-4.15.0-140-generic-x86_64-with-debian-stretch-sid
Python: 3.7.4 (default, Aug 13 2019, 20:35:49) [GCC 7.3.0]
Paddle compiled with cuda: True
NVCC: Build cuda_11.2.r11.2/compiler.29618528_0
cudnn: 8.2
GPUs used: 1
CUDA_VISIBLE_DEVICES: None
GPU: ['GPU 0: Tesla V100-SXM2-32GB']
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~16.04) 7.5.0
PaddlePaddle: 2.3.2
------------------------------------------------
/home/aistudio/MedicalSeg/medicalseg/cvlibs/config.py:451: UserWarning: Warning: The data dir now is /home/aistudio/MedicalSeg/data/, you should change the data_root in the global.yml if this directory didn't have enough space
.format(absolute_data_dir))
2022-09-07 11:49:50 [INFO]
---------------Config Information---------------
batch_size: 24
data_root: data/
export:
inference_helper:
type: TransUNetInferenceHelper
transforms:
- size:
- 1
- 224
- 224
type: Resize3D
iters: 13950
loss:
coef:
- 1
types:
- coef:
- 1
- 1
losses:
- type: CrossEntropyLoss
weight: null
- type: DiceLoss
type: MixedLoss
lr_scheduler:
decay_steps: 13950
end_lr: 0
learning_rate: 0.01
power: 0.9
type: PolynomialDecay
model:
attention_dropout_rate: 0.0
backbone:
block_units:
- 3
- 4
- 9
type: ResNet
width_factor: 1
classifier: seg
decoder_channels:
- 256
- 128
- 64
- 16
dropout_rate: 0.1
hidden_size: 768
img_size: 224
mlp_dim: 3072
n_skip: 3
num_classes: 9
num_heads: 12
num_layers: 12
patches_grid:
- 14
- 14
pretrained_path: https://paddleseg.bj.bcebos.com/paddleseg3d/synapse/transunet_synapse_1_224_224_14k_1e-2/pretrain_model.pdparams
skip_channels:
- 512
- 256
- 64
- 16
type: TransUNet
optimizer:
momentum: 0.9
type: sgd
weight_decay: 0.0001
test_dataset:
dataset_root: ./Synapse_npy
mode: test
num_classes: 9
result_dir: ./output
transforms:
- size:
- 1
- 224
- 224
type: Resize3D
type: Synapse
train_dataset:
dataset_root: ./Synapse_npy
mode: train
num_classes: 9
result_dir: ./output
transforms:
- flip_axis:
- 1
- 2
rotate_planes:
- - 1
- 2
type: RandomFlipRotation3D
- degrees: 20
prob: 0.5
rotate_planes:
- - 1
- 2
type: RandomRotation3D
- size:
- 1
- 224
- 224
type: Resize3D
type: Synapse
val_dataset:
dataset_root: ./Synapse_npy
mode: test
num_classes: 9
result_dir: ./output
transforms:
- size:
- 1
- 224
- 224
type: Resize3D
type: Synapse
------------------------------------------------
W0907 11:49:50.831183 5286 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
W0907 11:49:50.831221 5286 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
2022-09-07 11:49:51 [INFO] Loading pretrained model from https://paddleseg.bj.bcebos.com/paddleseg3d/synapse/transunet_synapse_1_224_224_14k_1e-2/pretrain_model.pdparams
Connecting to https://paddleseg.bj.bcebos.com/paddleseg3d/synapse/transunet_synapse_1_224_224_14k_1e-2/pretrain_model.pdparams
Downloading pretrain_model.pdparams
[ ] 0.00%[ ] 0.16%[ ] 0.60%[ ] 1.19%[= ] 2.05%[= ] 3.03%[= ] 3.92%[== ] 4.90%[=== ] 6.10%[=== ] 7.42%[==== ] 8.82%[===== ] 10.22%[===== ] 11.55%[====== ] 12.93%[======= ] 14.17%[======= ] 15.18%[======== ] 16.24%[======== ] 17.30%[========= ] 18.33%[========= ] 19.35%[========== ] 20.34%[========== ] 21.40%[=========== ] 22.32%[=========== ] 23.31%[============ ] 24.23%[============ ] 25.08%[============= ] 26.03%[============= ] 26.96%[============= ] 28.00%[============== ] 29.00%[=============== ] 30.10%[=============== ] 31.25%[================ ] 32.43%[================ ] 33.67%[================= ] 34.84%[================== ] 36.11%[================== ] 37.66%[=================== ] 38.93%[==================== ] 40.31%[==================== ] 41.74%[===================== ] 43.12%[====================== ] 44.50%[====================== ] 45.87%[======================= ] 47.59%[======================== ] 49.11%[========================= ] 50.54%[========================= ] 51.73%[========================== ] 52.98%[=========================== ] 54.40%[=========================== ] 55.82%[============================ ] 57.21%[============================= ] 58.70%[============================== ] 60.16%[============================== ] 61.54%[=============================== ] 62.74%[================================ ] 64.09%[================================ ] 65.24%[================================= ] 66.42%[================================= ] 67.46%[================================== ] 68.46%[================================== ] 69.53%[=================================== ] 70.66%[=================================== ] 71.82%[==================================== ] 73.01%[===================================== ] 74.10%[===================================== ] 75.30%[====================================== ] 76.68%[====================================== ] 77.85%[======================================= ] 78.90%[======================================= ] 79.97%[======================================== ] 80.88%[======================================== ] 81.95%[========================================= ] 83.02%[========================================== ] 84.04%[========================================== ] 85.11%[=========================================== ] 86.33%[=========================================== ] 87.73%[============================================ ] 88.92%[============================================ ] 89.91%[============================================= ] 90.96%[============================================== ] 92.04%[============================================== ] 93.08%[=============================================== ] 94.13%[=============================================== ] 95.17%[================================================ ] 96.38%[================================================ ] 97.23%[================================================= ] 98.31%[================================================= ] 99.21%[==================================================] 100.00%
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.0.attn.query.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.0.attn.query.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.0.attn.key.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.0.attn.key.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.0.attn.value.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.0.attn.value.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.0.attn.out.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.0.attn.out.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.1.attn.query.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.1.attn.query.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.1.attn.key.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.1.attn.key.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.1.attn.value.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.1.attn.value.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.1.attn.out.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.1.attn.out.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.2.attn.query.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.2.attn.query.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.2.attn.key.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.2.attn.key.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.2.attn.value.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.2.attn.value.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.2.attn.out.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.2.attn.out.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.3.attn.query.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.3.attn.query.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.3.attn.key.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.3.attn.key.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.3.attn.value.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.3.attn.value.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.3.attn.out.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.3.attn.out.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.4.attn.query.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.4.attn.query.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.4.attn.key.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.4.attn.key.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.4.attn.value.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.4.attn.value.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.4.attn.out.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.4.attn.out.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.5.attn.query.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.5.attn.query.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.5.attn.key.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.5.attn.key.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.5.attn.value.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.5.attn.value.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.5.attn.out.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.5.attn.out.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.6.attn.query.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.6.attn.query.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.6.attn.key.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.6.attn.key.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.6.attn.value.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.6.attn.value.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.6.attn.out.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.6.attn.out.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.7.attn.query.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.7.attn.query.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.7.attn.key.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.7.attn.key.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.7.attn.value.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.7.attn.value.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.7.attn.out.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.7.attn.out.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.8.attn.query.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.8.attn.query.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.8.attn.key.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.8.attn.key.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.8.attn.value.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.8.attn.value.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.8.attn.out.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.8.attn.out.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.9.attn.query.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.9.attn.query.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.9.attn.key.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.9.attn.key.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.9.attn.value.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.9.attn.value.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.9.attn.out.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.9.attn.out.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.10.attn.query.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.10.attn.query.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.10.attn.key.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.10.attn.key.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.10.attn.value.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.10.attn.value.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.10.attn.out.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.10.attn.out.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.11.attn.query.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.11.attn.query.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.11.attn.key.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.11.attn.key.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.11.attn.value.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.11.attn.value.bias is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.11.attn.out.weight is not in pretrained model
2022-09-07 11:50:01 [WARNING] transformer.encoder.layer.11.attn.out.bias is not in pretrained model
2022-09-07 11:50:01 [INFO] There are 304/400 variables loaded into TransUNet.
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/math_op_patch.py:278: UserWarning: The dtype of left and right variables are not the same, left dtype is paddle.float32, but right dtype is paddle.bool, the right dtype will convert to paddle.float32
format(lhs_dtype, rhs_dtype, lhs_dtype))
2022-09-07 11:50:09 [INFO] [TRAIN] epoch: 0, iter: 10/13950, loss: 2.4945, DSC: 11.4800, lr: 0.009994, batch_cost: 0.7757, reader_cost: 0.09192, ips: 30.9378 samples/sec | ETA 03:00:13
--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
0 paddle::platform::is_mlu_place(phi::Place const&)
----------------------
Error Message Summary:
----------------------
FatalError: `Termination signal` is detected by the operating system.
[TimeInfo: *** Aborted at 1662522612 (unix time) try "date -d @1662522612" if you are using GNU date ***]
[SignalInfo: *** SIGTERM (@0x3e800001524) received by PID 5286 (TID 0x7f05d47e0700) from PID 5412 ***]
2022-09-07 11:50:33 [INFO]
------------Environment Information-------------
platform: Linux-4.15.0-140-generic-x86_64-with-debian-stretch-sid
Python: 3.7.4 (default, Aug 13 2019, 20:35:49) [GCC 7.3.0]
Paddle compiled with cuda: True
NVCC: Build cuda_11.2.r11.2/compiler.29618528_0
cudnn: 8.2
GPUs used: 1
CUDA_VISIBLE_DEVICES: None
GPU: ['GPU 0: Tesla V100-SXM2-32GB']
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~16.04) 7.5.0
PaddlePaddle: 2.3.2
------------------------------------------------
/home/aistudio/MedicalSeg/medicalseg/cvlibs/config.py:451: UserWarning: Warning: The data dir now is /home/aistudio/MedicalSeg/data/, you should change the data_root in the global.yml if this directory didn't have enough space
.format(absolute_data_dir))
2022-09-07 11:50:33 [INFO]
---------------Config Information---------------
batch_size: 24
data_root: data/
export:
inference_helper:
type: TransUNetInferenceHelper
transforms:
- size:
- 1
- 224
- 224
type: Resize3D
iters: 13950
loss:
coef:
- 1
types:
- coef:
- 1
- 1
losses:
- type: CrossEntropyLoss
weight: null
- type: DiceLoss
type: MixedLoss
lr_scheduler:
decay_steps: 13950
end_lr: 0
learning_rate: 0.01
power: 0.9
type: PolynomialDecay
model:
attention_dropout_rate: 0.0
backbone:
block_units:
- 3
- 4
- 9
type: ResNet
width_factor: 1
classifier: seg
decoder_channels:
- 256
- 128
- 64
- 16
dropout_rate: 0.1
hidden_size: 768
img_size: 224
mlp_dim: 3072
n_skip: 3
num_classes: 9
num_heads: 12
num_layers: 12
patches_grid:
- 14
- 14
pretrained_path: https://paddleseg.bj.bcebos.com/paddleseg3d/synapse/transunet_synapse_1_224_224_14k_1e-2/pretrain_model.pdparams
skip_channels:
- 512
- 256
- 64
- 16
type: TransUNet
optimizer:
momentum: 0.9
type: sgd
weight_decay: 0.0001
test_dataset:
dataset_root: ./Synapse_npy
mode: test
num_classes: 9
result_dir: ./output
transforms:
- size:
- 1
- 224
- 224
type: Resize3D
type: Synapse
train_dataset:
dataset_root: ./Synapse_npy
mode: train
num_classes: 9
result_dir: ./output
transforms:
- flip_axis:
- 1
- 2
rotate_planes:
- - 1
- 2
type: RandomFlipRotation3D
- degrees: 20
prob: 0.5
rotate_planes:
- - 1
- 2
type: RandomRotation3D
- size:
- 1
- 224
- 224
type: Resize3D
type: Synapse
val_dataset:
dataset_root: ./Synapse_npy
mode: test
num_classes: 9
result_dir: ./output
transforms:
- size:
- 1
- 224
- 224
type: Resize3D
type: Synapse
------------------------------------------------
W0907 11:50:33.258643 5477 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
W0907 11:50:33.258687 5477 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
2022-09-07 11:50:34 [INFO] Loading pretrained model from https://paddleseg.bj.bcebos.com/paddleseg3d/synapse/transunet_synapse_1_224_224_14k_1e-2/pretrain_model.pdparams
Connecting to https://paddleseg.bj.bcebos.com/paddleseg3d/synapse/transunet_synapse_1_224_224_14k_1e-2/pretrain_model.pdparams
Downloading pretrain_model.pdparams
[ ] 0.00%[ ] 0.06%[ ] 0.12%[ ] 0.20%[ ] 0.29%[ ] 0.38%[ ] 0.48%[ ] 0.60%[ ] 0.72%[ ] 0.85%[ ] 1.00%[ ] 1.16%[ ] 1.34%[ ] 1.52%[ ] 1.70%[ ] 1.88%[= ] 2.05%[= ] 2.23%[= ] 2.44%[= ] 2.66%[= ] 2.90%[= ] 3.15%[= ] 3.41%[= ] 3.67%[= ] 3.94%[== ] 4.21%[== ] 4.47%[== ] 4.75%[== ] 4.99%[== ] 5.26%[== ] 5.56%[== ] 5.92%[=== ] 6.29%[=== ] 6.67%[=== ] 7.09%[=== ] 7.57%[==== ] 8.06%[==== ] 8.54%[==== ] 9.05%[==== ] 9.57%[===== ] 10.10%[===== ] 10.62%[===== ] 11.13%[===== ] 11.62%[====== ] 12.11%[====== ] 12.63%[====== ] 13.17%[====== ] 13.72%[======= ] 14.27%[======= ] 14.86%[======= ] 15.51%[======== ] 16.18%[======== ] 16.88%[======== ] 17.58%[========= ] 18.30%[========= ] 19.09%[========= ] 19.81%[========== ] 20.56%[========== ] 21.26%[=========== ] 22.03%[=========== ] 22.86%[=========== ] 23.72%[============ ] 24.56%[============ ] 25.37%[============= ] 26.25%[============= ] 27.19%[============== ] 28.05%[============== ] 29.03%[============== ] 29.96%[=============== ] 30.94%[=============== ] 31.91%[================ ] 32.83%[================ ] 33.75%[================= ] 34.84%[================= ] 35.97%[================== ] 37.19%[=================== ] 38.28%[=================== ] 39.40%[==================== ] 40.31%[==================== ] 41.36%[===================== ] 42.47%[===================== ] 43.44%[====================== ] 44.41%[====================== ] 45.39%[======================= ] 46.33%[======================= ] 47.37%[======================== ] 48.33%[======================== ] 49.15%[======================== ] 49.92%[========================= ] 50.80%[========================= ] 51.66%[========================== ] 52.59%[========================== ] 53.54%[=========================== ] 54.45%[=========================== ] 55.34%[============================ ] 56.25%[============================ ] 57.20%[============================= ] 58.19%[============================= ] 59.14%[============================== ] 60.22%[============================== ] 61.20%[=============================== ] 62.23%[=============================== ] 63.31%[================================ ] 64.43%[================================ ] 65.52%[================================= ] 66.71%[================================= ] 67.73%[================================== ] 68.86%[=================================== ] 70.07%[=================================== ] 71.13%[==================================== ] 72.44%[==================================== ] 73.69%[===================================== ] 74.99%[====================================== ] 76.11%[====================================== ] 76.96%[====================================== ] 77.90%[======================================= ] 78.91%[======================================= ] 80.00%[======================================== ] 81.07%[========================================= ] 82.02%[========================================= ] 82.82%[========================================= ] 83.93%[========================================== ] 85.10%[=========================================== ] 86.36%[=========================================== ] 87.48%[============================================ ] 88.59%[============================================ ] 89.53%[============================================= ] 90.72%[============================================= ] 91.89%[============================================== ] 93.07%[=============================================== ] 94.41%[=============================================== ] 95.60%[================================================ ] 96.55%[================================================ ] 97.43%[================================================= ] 98.54%[================================================= ] 99.57%[==================================================] 100.00%
2022-09-07 11:50:49 [INFO] There are 400/400 variables loaded into TransUNet.
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/math_op_patch.py:278: UserWarning: The dtype of left and right variables are not the same, left dtype is paddle.float32, but right dtype is paddle.bool, the right dtype will convert to paddle.float32
format(lhs_dtype, rhs_dtype, lhs_dtype))
2022-09-07 11:50:57 [INFO] [TRAIN] epoch: 0, iter: 10/13950, loss: 2.5018, DSC: 11.4480, lr: 0.009994, batch_cost: 0.8110, reader_cost: 0.12933, ips: 29.5933 samples/sec | ETA 03:08:25
2022-09-07 11:51:02 [INFO] [TRAIN] epoch: 0, iter: 20/13950, loss: 0.9948, DSC: 12.8828, lr: 0.009988, batch_cost: 0.4928, reader_cost: 0.00027, ips: 48.7046 samples/sec | ETA 01:54:24
2022-09-07 11:51:07 [INFO] [TRAIN] epoch: 0, iter: 30/13950, loss: 0.9925, DSC: 13.7872, lr: 0.009981, batch_cost: 0.4940, reader_cost: 0.00026, ips: 48.5790 samples/sec | ETA 01:54:37
2022-09-07 11:51:11 [INFO] [TRAIN] epoch: 0, iter: 40/13950, loss: 0.9541, DSC: 14.6018, lr: 0.009975, batch_cost: 0.4946, reader_cost: 0.00027, ips: 48.5214 samples/sec | ETA 01:54:40
2022-09-07 11:51:16 [INFO] [TRAIN] epoch: 0, iter: 50/13950, loss: 0.9311, DSC: 14.8355, lr: 0.009968, batch_cost: 0.4947, reader_cost: 0.00028, ips: 48.5188 samples/sec | ETA 01:54:35
2022-09-07 11:51:21 [INFO] [TRAIN] epoch: 0, iter: 60/13950, loss: 0.9165, DSC: 15.6598, lr: 0.009962, batch_cost: 0.4951, reader_cost: 0.00026, ips: 48.4786 samples/sec | ETA 01:54:36
2022-09-07 11:51:26 [INFO] [TRAIN] epoch: 0, iter: 70/13950, loss: 0.9027, DSC: 16.5057, lr: 0.009955, batch_cost: 0.4959, reader_cost: 0.00026, ips: 48.3959 samples/sec | ETA 01:54:43
2022-09-07 11:51:31 [INFO] [TRAIN] epoch: 0, iter: 80/13950, loss: 0.8899, DSC: 18.0665, lr: 0.009949, batch_cost: 0.4963, reader_cost: 0.00025, ips: 48.3613 samples/sec | ETA 01:54:43
2022-09-07 11:51:36 [INFO] [TRAIN] epoch: 0, iter: 90/13950, loss: 0.8772, DSC: 19.1904, lr: 0.009943, batch_cost: 0.4948, reader_cost: 0.00028, ips: 48.5020 samples/sec | ETA 01:54:18
2022-09-07 11:51:42 [INFO] [TRAIN] epoch: 1, iter: 100/13950, loss: 0.8661, DSC: 19.1731, lr: 0.009936, batch_cost: 0.5992, reader_cost: 0.12098, ips: 40.0554 samples/sec | ETA 02:18:18
2022-09-07 11:51:47 [INFO] [TRAIN] epoch: 1, iter: 110/13950, loss: 0.8621, DSC: 20.1777, lr: 0.009930, batch_cost: 0.4977, reader_cost: 0.00027, ips: 48.2258 samples/sec | ETA 01:54:47
2022-09-07 11:51:52 [INFO] [TRAIN] epoch: 1, iter: 120/13950, loss: 0.8506, DSC: 21.8082, lr: 0.009923, batch_cost: 0.4965, reader_cost: 0.00027, ips: 48.3390 samples/sec | ETA 01:54:26
2022-09-07 11:51:57 [INFO] [TRAIN] epoch: 1, iter: 130/13950, loss: 0.8363, DSC: 22.1735, lr: 0.009917, batch_cost: 0.5004, reader_cost: 0.00030, ips: 47.9573 samples/sec | ETA 01:55:16
2022-09-07 11:52:02 [INFO] [TRAIN] epoch: 1, iter: 140/13950, loss: 0.8356, DSC: 22.6151, lr: 0.009910, batch_cost: 0.4982, reader_cost: 0.00031, ips: 48.1728 samples/sec | ETA 01:54:40
2022-09-07 11:52:07 [INFO] [TRAIN] epoch: 1, iter: 150/13950, loss: 0.8285, DSC: 24.3006, lr: 0.009904, batch_cost: 0.5015, reader_cost: 0.00030, ips: 47.8522 samples/sec | ETA 01:55:21
2022-09-07 11:52:12 [INFO] [TRAIN] epoch: 1, iter: 160/13950, loss: 0.8174, DSC: 24.4328, lr: 0.009897, batch_cost: 0.5004, reader_cost: 0.00029, ips: 47.9643 samples/sec | ETA 01:55:00
2022-09-07 11:52:17 [INFO] [TRAIN] epoch: 1, iter: 170/13950, loss: 0.8120, DSC: 25.3727, lr: 0.009891, batch_cost: 0.5036, reader_cost: 0.00029, ips: 47.6584 samples/sec | ETA 01:55:39
2022-09-07 11:52:22 [INFO] [TRAIN] epoch: 1, iter: 180/13950, loss: 0.8018, DSC: 26.2308, lr: 0.009884, batch_cost: 0.4994, reader_cost: 0.00027, ips: 48.0544 samples/sec | ETA 01:54:37
2022-09-07 11:52:28 [INFO] [TRAIN] epoch: 2, iter: 190/13950, loss: 0.8127, DSC: 24.6344, lr: 0.009878, batch_cost: 0.6110, reader_cost: 0.12442, ips: 39.2809 samples/sec | ETA 02:20:07
2022-09-07 11:52:33 [INFO] [TRAIN] epoch: 2, iter: 200/13950, loss: 0.7944, DSC: 26.5479, lr: 0.009872, batch_cost: 0.5016, reader_cost: 0.00026, ips: 47.8492 samples/sec | ETA 01:54:56
2022-09-07 11:52:38 [INFO] [TRAIN] epoch: 2, iter: 210/13950, loss: 0.7806, DSC: 27.6409, lr: 0.009865, batch_cost: 0.4997, reader_cost: 0.00032, ips: 48.0288 samples/sec | ETA 01:54:25
2022-09-07 11:52:43 [INFO] [TRAIN] epoch: 2, iter: 220/13950, loss: 0.7756, DSC: 28.6790, lr: 0.009859, batch_cost: 0.4985, reader_cost: 0.00027, ips: 48.1402 samples/sec | ETA 01:54:05
2022-09-07 11:52:48 [INFO] [TRAIN] epoch: 2, iter: 230/13950, loss: 0.7756, DSC: 29.0652, lr: 0.009852, batch_cost: 0.5003, reader_cost: 0.00026, ips: 47.9732 samples/sec | ETA 01:54:23
2022-09-07 11:52:53 [INFO] [TRAIN] epoch: 2, iter: 240/13950, loss: 0.7692, DSC: 29.3559, lr: 0.009846, batch_cost: 0.5018, reader_cost: 0.00027, ips: 47.8235 samples/sec | ETA 01:54:40
2022-09-07 11:52:58 [INFO] [TRAIN] epoch: 2, iter: 250/13950, loss: 0.7572, DSC: 30.2228, lr: 0.009839, batch_cost: 0.5037, reader_cost: 0.00033, ips: 47.6495 samples/sec | ETA 01:55:00
2022-09-07 11:53:03 [INFO] [TRAIN] epoch: 2, iter: 260/13950, loss: 0.7452, DSC: 30.4972, lr: 0.009833, batch_cost: 0.5024, reader_cost: 0.00027, ips: 47.7722 samples/sec | ETA 01:54:37
2022-09-07 11:53:08 [INFO] [TRAIN] epoch: 2, iter: 270/13950, loss: 0.7513, DSC: 30.5025, lr: 0.009826, batch_cost: 0.5018, reader_cost: 0.00026, ips: 47.8289 samples/sec | ETA 01:54:24
2022-09-07 11:53:14 [INFO] [TRAIN] epoch: 3, iter: 280/13950, loss: 0.7416, DSC: 32.8919, lr: 0.009820, batch_cost: 0.5974, reader_cost: 0.11801, ips: 40.1720 samples/sec | ETA 02:16:06
2022-09-07 11:53:19 [INFO] [TRAIN] epoch: 3, iter: 290/13950, loss: 0.7263, DSC: 33.9826, lr: 0.009813, batch_cost: 0.5103, reader_cost: 0.00038, ips: 47.0357 samples/sec | ETA 01:56:10
2022-09-07 11:53:25 [INFO] [TRAIN] epoch: 3, iter: 300/13950, loss: 0.7042, DSC: 36.3885, lr: 0.009807, batch_cost: 0.5031, reader_cost: 0.00029, ips: 47.7034 samples/sec | ETA 01:54:27
2022-09-07 11:53:30 [INFO] [TRAIN] epoch: 3, iter: 310/13950, loss: 0.6932, DSC: 36.4719, lr: 0.009800, batch_cost: 0.5041, reader_cost: 0.00034, ips: 47.6077 samples/sec | ETA 01:54:36
2022-09-07 11:53:35 [INFO] [TRAIN] epoch: 3, iter: 320/13950, loss: 0.6791, DSC: 37.8334, lr: 0.009794, batch_cost: 0.5043, reader_cost: 0.00039, ips: 47.5940 samples/sec | ETA 01:54:33
2022-09-07 11:53:40 [INFO] [TRAIN] epoch: 3, iter: 330/13950, loss: 0.6575, DSC: 39.7633, lr: 0.009787, batch_cost: 0.5042, reader_cost: 0.00027, ips: 47.6030 samples/sec | ETA 01:54:26
2022-09-07 11:53:45 [INFO] [TRAIN] epoch: 3, iter: 340/13950, loss: 0.6361, DSC: 41.6647, lr: 0.009781, batch_cost: 0.5046, reader_cost: 0.00028, ips: 47.5645 samples/sec | ETA 01:54:27
2022-09-07 11:53:50 [INFO] [TRAIN] epoch: 3, iter: 350/13950, loss: 0.6290, DSC: 43.0757, lr: 0.009775, batch_cost: 0.5069, reader_cost: 0.00029, ips: 47.3472 samples/sec | ETA 01:54:53
2022-09-07 11:53:55 [INFO] [TRAIN] epoch: 3, iter: 360/13950, loss: 0.6129, DSC: 43.0695, lr: 0.009768, batch_cost: 0.5058, reader_cost: 0.00028, ips: 47.4473 samples/sec | ETA 01:54:34
2022-09-07 11:54:00 [INFO] [TRAIN] epoch: 3, iter: 370/13950, loss: 0.5984, DSC: 45.5799, lr: 0.009762, batch_cost: 0.5028, reader_cost: 0.00032, ips: 47.7315 samples/sec | ETA 01:53:48
2022-09-07 11:54:06 [INFO] [TRAIN] epoch: 4, iter: 380/13950, loss: 0.6024, DSC: 43.9555, lr: 0.009755, batch_cost: 0.6012, reader_cost: 0.11265, ips: 39.9200 samples/sec | ETA 02:15:58
2022-09-07 11:54:11 [INFO] [TRAIN] epoch: 4, iter: 390/13950, loss: 0.6332, DSC: 42.3259, lr: 0.009749, batch_cost: 0.5082, reader_cost: 0.00030, ips: 47.2270 samples/sec | ETA 01:54:50
2022-09-07 11:54:16 [INFO] [TRAIN] epoch: 4, iter: 400/13950, loss: 0.5888, DSC: 45.7674, lr: 0.009742, batch_cost: 0.5057, reader_cost: 0.00034, ips: 47.4583 samples/sec | ETA 01:54:12
2022-09-07 11:54:21 [INFO] [TRAIN] epoch: 4, iter: 410/13950, loss: 0.5623, DSC: 48.9231, lr: 0.009736, batch_cost: 0.5056, reader_cost: 0.00028, ips: 47.4698 samples/sec | ETA 01:54:05
2022-09-07 11:54:26 [INFO] [TRAIN] epoch: 4, iter: 420/13950, loss: 0.5327, DSC: 51.2247, lr: 0.009729, batch_cost: 0.5057, reader_cost: 0.00029, ips: 47.4571 samples/sec | ETA 01:54:02
2022-09-07 11:54:31 [INFO] [TRAIN] epoch: 4, iter: 430/13950, loss: 0.4935, DSC: 56.0343, lr: 0.009723, batch_cost: 0.5048, reader_cost: 0.00027, ips: 47.5480 samples/sec | ETA 01:53:44
2022-09-07 11:54:36 [INFO] [TRAIN] epoch: 4, iter: 440/13950, loss: 0.4978, DSC: 55.0798, lr: 0.009716, batch_cost: 0.5047, reader_cost: 0.00038, ips: 47.5557 samples/sec | ETA 01:53:38
2022-09-07 11:54:41 [INFO] [TRAIN] epoch: 4, iter: 450/13950, loss: 0.4899, DSC: 55.5863, lr: 0.009710, batch_cost: 0.5049, reader_cost: 0.00027, ips: 47.5385 samples/sec | ETA 01:53:35
2022-09-07 11:54:46 [INFO] [TRAIN] epoch: 4, iter: 460/13950, loss: 0.4765, DSC: 56.5683, lr: 0.009703, batch_cost: 0.5031, reader_cost: 0.00025, ips: 47.7059 samples/sec | ETA 01:53:06
2022-09-07 11:54:52 [INFO] [TRAIN] epoch: 5, iter: 470/13950, loss: 0.4964, DSC: 54.3620, lr: 0.009697, batch_cost: 0.6102, reader_cost: 0.11932, ips: 39.3322 samples/sec | ETA 02:17:05
2022-09-07 11:54:57 [INFO] [TRAIN] epoch: 5, iter: 480/13950, loss: 0.4719, DSC: 57.1715, lr: 0.009690, batch_cost: 0.5041, reader_cost: 0.00030, ips: 47.6056 samples/sec | ETA 01:53:10
2022-09-07 11:55:02 [INFO] [TRAIN] epoch: 5, iter: 490/13950, loss: 0.4662, DSC: 57.5773, lr: 0.009684, batch_cost: 0.5024, reader_cost: 0.00028, ips: 47.7705 samples/sec | ETA 01:52:42
2022-09-07 11:55:07 [INFO] [TRAIN] epoch: 5, iter: 500/13950, loss: 0.4592, DSC: 58.1426, lr: 0.009677, batch_cost: 0.5026, reader_cost: 0.00028, ips: 47.7543 samples/sec | ETA 01:52:39
2022-09-07 11:55:13 [INFO] [TRAIN] epoch: 5, iter: 510/13950, loss: 0.4359, DSC: 59.7246, lr: 0.009671, batch_cost: 0.5065, reader_cost: 0.00028, ips: 47.3806 samples/sec | ETA 01:53:27
2022-09-07 11:55:18 [INFO] [TRAIN] epoch: 5, iter: 520/13950, loss: 0.4453, DSC: 59.4985, lr: 0.009665, batch_cost: 0.5032, reader_cost: 0.00034, ips: 47.6958 samples/sec | ETA 01:52:37
2022-09-07 11:55:23 [INFO] [TRAIN] epoch: 5, iter: 530/13950, loss: 0.4241, DSC: 61.3035, lr: 0.009658, batch_cost: 0.5037, reader_cost: 0.00029, ips: 47.6459 samples/sec | ETA 01:52:39
2022-09-07 11:55:28 [INFO] [TRAIN] epoch: 5, iter: 540/13950, loss: 0.4214, DSC: 61.0743, lr: 0.009652, batch_cost: 0.5048, reader_cost: 0.00034, ips: 47.5481 samples/sec | ETA 01:52:48
2022-09-07 11:55:33 [INFO] [TRAIN] epoch: 5, iter: 550/13950, loss: 0.4103, DSC: 62.6077, lr: 0.009645, batch_cost: 0.5059, reader_cost: 0.00027, ips: 47.4395 samples/sec | ETA 01:52:59
2022-09-07 11:55:39 [INFO] [TRAIN] epoch: 6, iter: 560/13950, loss: 0.3965, DSC: 63.9007, lr: 0.009639, batch_cost: 0.6055, reader_cost: 0.12321, ips: 39.6369 samples/sec | ETA 02:15:07
2022-09-07 11:55:44 [INFO] [TRAIN] epoch: 6, iter: 570/13950, loss: 0.3997, DSC: 63.5642, lr: 0.009632, batch_cost: 0.5048, reader_cost: 0.00029, ips: 47.5463 samples/sec | ETA 01:52:33
2022-09-07 11:55:49 [INFO] [TRAIN] epoch: 6, iter: 580/13950, loss: 0.3646, DSC: 66.6703, lr: 0.009626, batch_cost: 0.5025, reader_cost: 0.00028, ips: 47.7610 samples/sec | ETA 01:51:58
2022-09-07 11:55:54 [INFO] [TRAIN] epoch: 6, iter: 590/13950, loss: 0.3491, DSC: 68.3966, lr: 0.009619, batch_cost: 0.5045, reader_cost: 0.00028, ips: 47.5677 samples/sec | ETA 01:52:20
2022-09-07 11:55:59 [INFO] [TRAIN] epoch: 6, iter: 600/13950, loss: 0.3637, DSC: 66.8093, lr: 0.009613, batch_cost: 0.5047, reader_cost: 0.00029, ips: 47.5573 samples/sec | ETA 01:52:17
2022-09-07 11:56:04 [INFO] [TRAIN] epoch: 6, iter: 610/13950, loss: 0.3393, DSC: 69.3414, lr: 0.009606, batch_cost: 0.5035, reader_cost: 0.00028, ips: 47.6626 samples/sec | ETA 01:51:57
2022-09-07 11:56:09 [INFO] [TRAIN] epoch: 6, iter: 620/13950, loss: 0.3549, DSC: 67.6056, lr: 0.009600, batch_cost: 0.5036, reader_cost: 0.00027, ips: 47.6543 samples/sec | ETA 01:51:53
2022-09-07 11:56:14 [INFO] [TRAIN] epoch: 6, iter: 630/13950, loss: 0.3577, DSC: 67.4019, lr: 0.009593, batch_cost: 0.5045, reader_cost: 0.00028, ips: 47.5699 samples/sec | ETA 01:52:00
2022-09-07 11:56:19 [INFO] [TRAIN] epoch: 6, iter: 640/13950, loss: 0.3578, DSC: 67.4579, lr: 0.009587, batch_cost: 0.5033, reader_cost: 0.00026, ips: 47.6839 samples/sec | ETA 01:51:39
2022-09-07 11:56:24 [INFO] [TRAIN] epoch: 6, iter: 650/13950, loss: 0.3532, DSC: 67.7578, lr: 0.009580, batch_cost: 0.5036, reader_cost: 0.00023, ips: 47.6575 samples/sec | ETA 01:51:37
2022-09-07 11:56:30 [INFO] [TRAIN] epoch: 7, iter: 660/13950, loss: 0.3729, DSC: 66.7252, lr: 0.009574, batch_cost: 0.6133, reader_cost: 0.11197, ips: 39.1339 samples/sec | ETA 02:15:50
2022-09-07 11:56:35 [INFO] [TRAIN] epoch: 7, iter: 670/13950, loss: 0.3429, DSC: 69.2355, lr: 0.009567, batch_cost: 0.5046, reader_cost: 0.00028, ips: 47.5598 samples/sec | ETA 01:51:41
2022-09-07 11:56:40 [INFO] [TRAIN] epoch: 7, iter: 680/13950, loss: 0.3201, DSC: 70.8619, lr: 0.009561, batch_cost: 0.5034, reader_cost: 0.00027, ips: 47.6754 samples/sec | ETA 01:51:20
2022-09-07 11:56:45 [INFO] [TRAIN] epoch: 7, iter: 690/13950, loss: 0.3136, DSC: 71.1949, lr: 0.009554, batch_cost: 0.5037, reader_cost: 0.00029, ips: 47.6479 samples/sec | ETA 01:51:18
2022-09-07 11:56:50 [INFO] [TRAIN] epoch: 7, iter: 700/13950, loss: 0.3369, DSC: 68.7318, lr: 0.009548, batch_cost: 0.5047, reader_cost: 0.00028, ips: 47.5532 samples/sec | ETA 01:51:27
2022-09-07 11:56:55 [INFO] [TRAIN] epoch: 7, iter: 710/13950, loss: 0.3254, DSC: 69.8315, lr: 0.009541, batch_cost: 0.5050, reader_cost: 0.00029, ips: 47.5288 samples/sec | ETA 01:51:25
2022-09-07 11:57:01 [INFO] [TRAIN] epoch: 7, iter: 720/13950, loss: 0.3022, DSC: 72.4038, lr: 0.009535, batch_cost: 0.5068, reader_cost: 0.00031, ips: 47.3552 samples/sec | ETA 01:51:45
2022-09-07 11:57:06 [INFO] [TRAIN] epoch: 7, iter: 730/13950, loss: 0.2863, DSC: 73.9581, lr: 0.009528, batch_cost: 0.5052, reader_cost: 0.00030, ips: 47.5097 samples/sec | ETA 01:51:18
2022-09-07 11:57:11 [INFO] [TRAIN] epoch: 7, iter: 740/13950, loss: 0.2776, DSC: 74.7168, lr: 0.009522, batch_cost: 0.5039, reader_cost: 0.00027, ips: 47.6266 samples/sec | ETA 01:50:56
2022-09-07 11:57:17 [INFO] [TRAIN] epoch: 8, iter: 750/13950, loss: 0.3190, DSC: 70.6327, lr: 0.009515, batch_cost: 0.6291, reader_cost: 0.13135, ips: 38.1473 samples/sec | ETA 02:18:24
2022-09-07 11:57:22 [INFO] [TRAIN] epoch: 8, iter: 760/13950, loss: 0.2688, DSC: 75.6918, lr: 0.009509, batch_cost: 0.5068, reader_cost: 0.00030, ips: 47.3546 samples/sec | ETA 01:51:24
2022-09-07 11:57:27 [INFO] [TRAIN] epoch: 8, iter: 770/13950, loss: 0.2706, DSC: 75.2449, lr: 0.009502, batch_cost: 0.5063, reader_cost: 0.00030, ips: 47.4026 samples/sec | ETA 01:51:13
2022-09-07 11:57:32 [INFO] [TRAIN] epoch: 8, iter: 780/13950, loss: 0.2925, DSC: 73.1948, lr: 0.009496, batch_cost: 0.5070, reader_cost: 0.00030, ips: 47.3392 samples/sec | ETA 01:51:16
2022-09-07 11:57:37 [INFO] [TRAIN] epoch: 8, iter: 790/13950, loss: 0.2564, DSC: 76.3509, lr: 0.009489, batch_cost: 0.5078, reader_cost: 0.00038, ips: 47.2582 samples/sec | ETA 01:51:23
2022-09-07 11:57:42 [INFO] [TRAIN] epoch: 8, iter: 800/13950, loss: 0.2898, DSC: 73.3192, lr: 0.009483, batch_cost: 0.5051, reader_cost: 0.00029, ips: 47.5121 samples/sec | ETA 01:50:42
2022-09-07 11:57:47 [INFO] [TRAIN] epoch: 8, iter: 810/13950, loss: 0.2769, DSC: 74.8726, lr: 0.009477, batch_cost: 0.5055, reader_cost: 0.00029, ips: 47.4819 samples/sec | ETA 01:50:41
2022-09-07 11:57:52 [INFO] [TRAIN] epoch: 8, iter: 820/13950, loss: 0.2858, DSC: 73.7668, lr: 0.009470, batch_cost: 0.5057, reader_cost: 0.00036, ips: 47.4604 samples/sec | ETA 01:50:39
2022-09-07 11:57:57 [INFO] [TRAIN] epoch: 8, iter: 830/13950, loss: 0.2615, DSC: 76.3177, lr: 0.009464, batch_cost: 0.5071, reader_cost: 0.00039, ips: 47.3289 samples/sec | ETA 01:50:53
2022-09-07 11:58:04 [INFO] [TRAIN] epoch: 9, iter: 840/13950, loss: 0.2913, DSC: 73.4651, lr: 0.009457, batch_cost: 0.6148, reader_cost: 0.12114, ips: 39.0399 samples/sec | ETA 02:14:19
2022-09-07 11:58:09 [INFO] [TRAIN] epoch: 9, iter: 850/13950, loss: 0.2906, DSC: 73.4650, lr: 0.009451, batch_cost: 0.5108, reader_cost: 0.00036, ips: 46.9823 samples/sec | ETA 01:51:31
2022-09-07 11:58:14 [INFO] [TRAIN] epoch: 9, iter: 860/13950, loss: 0.2599, DSC: 76.0388, lr: 0.009444, batch_cost: 0.5029, reader_cost: 0.00028, ips: 47.7221 samples/sec | ETA 01:49:43
2022-09-07 11:58:19 [INFO] [TRAIN] epoch: 9, iter: 870/13950, loss: 0.2638, DSC: 76.1200, lr: 0.009438, batch_cost: 0.5062, reader_cost: 0.00029, ips: 47.4122 samples/sec | ETA 01:50:21
2022-09-07 11:58:24 [INFO] [TRAIN] epoch: 9, iter: 880/13950, loss: 0.2577, DSC: 76.6586, lr: 0.009431, batch_cost: 0.5022, reader_cost: 0.00029, ips: 47.7938 samples/sec | ETA 01:49:23
2022-09-07 11:58:29 [INFO] [TRAIN] epoch: 9, iter: 890/13950, loss: 0.2296, DSC: 79.1241, lr: 0.009425, batch_cost: 0.5041, reader_cost: 0.00028, ips: 47.6051 samples/sec | ETA 01:49:44
2022-09-07 11:58:34 [INFO] [TRAIN] epoch: 9, iter: 900/13950, loss: 0.2560, DSC: 76.2506, lr: 0.009418, batch_cost: 0.5044, reader_cost: 0.00029, ips: 47.5770 samples/sec | ETA 01:49:43
2022-09-07 11:58:39 [INFO] [TRAIN] epoch: 9, iter: 910/13950, loss: 0.2559, DSC: 76.5617, lr: 0.009412, batch_cost: 0.5073, reader_cost: 0.00031, ips: 47.3107 samples/sec | ETA 01:50:14
2022-09-07 11:58:44 [INFO] [TRAIN] epoch: 9, iter: 920/13950, loss: 0.2678, DSC: 75.3316, lr: 0.009405, batch_cost: 0.5050, reader_cost: 0.00034, ips: 47.5248 samples/sec | ETA 01:49:40
2022-09-07 11:58:49 [INFO] [TRAIN] epoch: 10, iter: 930/13950, loss: 0.2660, DSC: 75.4568, lr: 0.009399, batch_cost: 0.4765, reader_cost: 0.00025, ips: 50.3663 samples/sec | ETA 01:43:24
2022-09-07 11:58:55 [INFO] [TRAIN] epoch: 10, iter: 940/13950, loss: 0.2424, DSC: 77.8399, lr: 0.009392, batch_cost: 0.6463, reader_cost: 0.12780, ips: 37.1362 samples/sec | ETA 02:20:07
2022-09-07 11:59:00 [INFO] [TRAIN] epoch: 10, iter: 950/13950, loss: 0.2381, DSC: 78.5076, lr: 0.009386, batch_cost: 0.5068, reader_cost: 0.00029, ips: 47.3597 samples/sec | ETA 01:49:47
2022-09-07 11:59:05 [INFO] [TRAIN] epoch: 10, iter: 960/13950, loss: 0.2459, DSC: 77.1455, lr: 0.009379, batch_cost: 0.5061, reader_cost: 0.00036, ips: 47.4199 samples/sec | ETA 01:49:34
2022-09-07 11:59:10 [INFO] [TRAIN] epoch: 10, iter: 970/13950, loss: 0.2588, DSC: 76.1731, lr: 0.009373, batch_cost: 0.5074, reader_cost: 0.00031, ips: 47.2980 samples/sec | ETA 01:49:46
2022-09-07 11:59:16 [INFO] [TRAIN] epoch: 10, iter: 980/13950, loss: 0.2349, DSC: 78.8821, lr: 0.009366, batch_cost: 0.5047, reader_cost: 0.00036, ips: 47.5523 samples/sec | ETA 01:49:06
2022-09-07 11:59:21 [INFO] [TRAIN] epoch: 10, iter: 990/13950, loss: 0.2063, DSC: 81.0452, lr: 0.009360, batch_cost: 0.5074, reader_cost: 0.00029, ips: 47.3003 samples/sec | ETA 01:49:35
2022-09-07 11:59:26 [INFO] [TRAIN] epoch: 10, iter: 1000/13950, loss: 0.2312, DSC: 78.9193, lr: 0.009353, batch_cost: 0.5043, reader_cost: 0.00029, ips: 47.5892 samples/sec | ETA 01:48:50
2022-09-07 11:59:26 [INFO] Start evaluating (total_samples: 12, total_iters: 12)...
1/12 [=>............................] - ETA: 22s - batch_cost: 2.0030 - reader cost: 1.1533 2/12 [====>.........................] - ETA: 12s - batch_cost: 1.2683 - reader cost: 0.5794 3/12 [======>.......................] - ETA: 9s - batch_cost: 1.0467 - reader cost: 0.3940 4/12 [=========>....................] - ETA: 9s - batch_cost: 1.1963 - reader cost: 0.4463 5/12 [===========>..................] - ETA: 7s - batch_cost: 1.1284 - reader cost: 0.3614 6/12 [==============>...............] - ETA: 6s - batch_cost: 1.0809 - reader cost: 0.3101 7/12 [================>.............] - ETA: 5s - batch_cost: 1.0122 - reader cost: 0.2686 8/12 [===================>..........] - ETA: 4s - batch_cost: 1.1133 - reader cost: 0.3180 9/12 [=====================>........] - ETA: 3s - batch_cost: 1.0875 - reader cost: 0.284910/12 [========================>.....] - ETA: 2s - batch_cost: 1.0625 - reader cost: 0.259511/12 [==========================>...] - ETA: 1s - batch_cost: 1.0119 - reader cost: 0.237412/12 [==============================] - 12s 975ms/step - batch_cost: 0.9750 - reader cost: 0.2194
2022-09-07 11:59:37 [INFO] [EVAL] #Images: 12, Dice: 0.7425, Loss: 0.287339
2022-09-07 11:59:37 [INFO] [EVAL] Class dice:
[0.9898 0.8298 0.3434 0.7943 0.7939 0.8981 0.5808 0.7938 0.6587]
2022-09-07 11:59:40 [INFO] [EVAL] The model with the best validation mDice (0.7425) was saved at iter 1000.
2022-09-07 11:59:46 [INFO] [TRAIN] epoch: 10, iter: 1010/13950, loss: 0.2385, DSC: 78.1162, lr: 0.009347, batch_cost: 0.5057, reader_cost: 0.00048, ips: 47.4633 samples/sec | ETA 01:49:03
2022-09-07 11:59:51 [INFO] [TRAIN] epoch: 10, iter: 1020/13950, loss: 0.2252, DSC: 79.2737, lr: 0.009340, batch_cost: 0.5048, reader_cost: 0.00035, ips: 47.5398 samples/sec | ETA 01:48:47
2022-09-07 11:59:57 [INFO] [TRAIN] epoch: 11, iter: 1030/13950, loss: 0.2500, DSC: 76.7184, lr: 0.009334, batch_cost: 0.6009, reader_cost: 0.11016, ips: 39.9423 samples/sec | ETA 02:09:23
2022-09-07 12:00:02 [INFO] [TRAIN] epoch: 11, iter: 1040/13950, loss: 0.2334, DSC: 78.5381, lr: 0.009327, batch_cost: 0.5027, reader_cost: 0.00030, ips: 47.7469 samples/sec | ETA 01:48:09
2022-09-07 12:00:07 [INFO] [TRAIN] epoch: 11, iter: 1050/13950, loss: 0.2418, DSC: 77.7909, lr: 0.009321, batch_cost: 0.5029, reader_cost: 0.00035, ips: 47.7262 samples/sec | ETA 01:48:07
2022-09-07 12:00:12 [INFO] [TRAIN] epoch: 11, iter: 1060/13950, loss: 0.2381, DSC: 77.9033, lr: 0.009314, batch_cost: 0.5032, reader_cost: 0.00036, ips: 47.6992 samples/sec | ETA 01:48:05
2022-09-07 12:00:17 [INFO] [TRAIN] epoch: 11, iter: 1070/13950, loss: 0.2337, DSC: 78.8707, lr: 0.009308, batch_cost: 0.5040, reader_cost: 0.00034, ips: 47.6213 samples/sec | ETA 01:48:11
2022-09-07 12:00:22 [INFO] [TRAIN] epoch: 11, iter: 1080/13950, loss: 0.2250, DSC: 79.3062, lr: 0.009301, batch_cost: 0.5083, reader_cost: 0.00038, ips: 47.2189 samples/sec | ETA 01:49:01
2022-09-07 12:00:27 [INFO] [TRAIN] epoch: 11, iter: 1090/13950, loss: 0.2347, DSC: 78.4494, lr: 0.009295, batch_cost: 0.5048, reader_cost: 0.00029, ips: 47.5440 samples/sec | ETA 01:48:11
2022-09-07 12:00:32 [INFO] [TRAIN] epoch: 11, iter: 1100/13950, loss: 0.2154, DSC: 80.1309, lr: 0.009288, batch_cost: 0.5041, reader_cost: 0.00030, ips: 47.6057 samples/sec | ETA 01:47:58
2022-09-07 12:00:37 [INFO] [TRAIN] epoch: 11, iter: 1110/13950, loss: 0.1977, DSC: 81.9830, lr: 0.009282, batch_cost: 0.5043, reader_cost: 0.00028, ips: 47.5885 samples/sec | ETA 01:47:55
2022-09-07 12:00:44 [INFO] [TRAIN] epoch: 12, iter: 1120/13950, loss: 0.2231, DSC: 79.5187, lr: 0.009275, batch_cost: 0.6628, reader_cost: 0.16849, ips: 36.2090 samples/sec | ETA 02:21:43
2022-09-07 12:00:49 [INFO] [TRAIN] epoch: 12, iter: 1130/13950, loss: 0.2180, DSC: 79.9861, lr: 0.009269, batch_cost: 0.5073, reader_cost: 0.00036, ips: 47.3079 samples/sec | ETA 01:48:23
2022-09-07 12:00:54 [INFO] [TRAIN] epoch: 12, iter: 1140/13950, loss: 0.2168, DSC: 79.7567, lr: 0.009262, batch_cost: 0.5074, reader_cost: 0.00030, ips: 47.2974 samples/sec | ETA 01:48:20
2022-09-07 12:00:59 [INFO] [TRAIN] epoch: 12, iter: 1150/13950, loss: 0.2443, DSC: 77.5324, lr: 0.009256, batch_cost: 0.5068, reader_cost: 0.00036, ips: 47.3591 samples/sec | ETA 01:48:06
2022-09-07 12:01:04 [INFO] [TRAIN] epoch: 12, iter: 1160/13950, loss: 0.2120, DSC: 80.5988, lr: 0.009249, batch_cost: 0.5068, reader_cost: 0.00036, ips: 47.3516 samples/sec | ETA 01:48:02
2022-09-07 12:01:09 [INFO] [TRAIN] epoch: 12, iter: 1170/13950, loss: 0.1986, DSC: 81.8853, lr: 0.009243, batch_cost: 0.5063, reader_cost: 0.00036, ips: 47.4016 samples/sec | ETA 01:47:50
2022-09-07 12:01:14 [INFO] [TRAIN] epoch: 12, iter: 1180/13950, loss: 0.1942, DSC: 82.2847, lr: 0.009236, batch_cost: 0.5074, reader_cost: 0.00030, ips: 47.2979 samples/sec | ETA 01:47:59
2022-09-07 12:01:19 [INFO] [TRAIN] epoch: 12, iter: 1190/13950, loss: 0.2066, DSC: 80.9038, lr: 0.009230, batch_cost: 0.5059, reader_cost: 0.00029, ips: 47.4413 samples/sec | ETA 01:47:35
2022-09-07 12:01:24 [INFO] [TRAIN] epoch: 12, iter: 1200/13950, loss: 0.1839, DSC: 83.3554, lr: 0.009223, batch_cost: 0.5073, reader_cost: 0.00030, ips: 47.3102 samples/sec | ETA 01:47:47
2022-09-07 12:01:30 [INFO] [TRAIN] epoch: 13, iter: 1210/13950, loss: 0.2102, DSC: 80.7673, lr: 0.009217, batch_cost: 0.5937, reader_cost: 0.11213, ips: 40.4243 samples/sec | ETA 02:06:03
2022-09-07 12:01:36 [INFO] [TRAIN] epoch: 13, iter: 1220/13950, loss: 0.1940, DSC: 82.3094, lr: 0.009210, batch_cost: 0.5223, reader_cost: 0.00037, ips: 45.9479 samples/sec | ETA 01:50:49
2022-09-07 12:01:41 [INFO] [TRAIN] epoch: 13, iter: 1230/13950, loss: 0.1995, DSC: 81.4175, lr: 0.009203, batch_cost: 0.5058, reader_cost: 0.00035, ips: 47.4462 samples/sec | ETA 01:47:14
2022-09-07 12:01:46 [INFO] [TRAIN] epoch: 13, iter: 1240/13950, loss: 0.2250, DSC: 78.9576, lr: 0.009197, batch_cost: 0.5147, reader_cost: 0.00029, ips: 46.6309 samples/sec | ETA 01:49:01
2022-09-07 12:01:51 [INFO] [TRAIN] epoch: 13, iter: 1250/13950, loss: 0.2119, DSC: 80.3331, lr: 0.009190, batch_cost: 0.5055, reader_cost: 0.00027, ips: 47.4731 samples/sec | ETA 01:47:00
2022-09-07 12:01:56 [INFO] [TRAIN] epoch: 13, iter: 1260/13950, loss: 0.2131, DSC: 80.6082, lr: 0.009184, batch_cost: 0.5065, reader_cost: 0.00028, ips: 47.3853 samples/sec | ETA 01:47:07
2022-09-07 12:02:01 [INFO] [TRAIN] epoch: 13, iter: 1270/13950, loss: 0.1979, DSC: 81.8700, lr: 0.009177, batch_cost: 0.5061, reader_cost: 0.00034, ips: 47.4228 samples/sec | ETA 01:46:57
2022-09-07 12:02:06 [INFO] [TRAIN] epoch: 13, iter: 1280/13950, loss: 0.1919, DSC: 82.4028, lr: 0.009171, batch_cost: 0.5055, reader_cost: 0.00029, ips: 47.4803 samples/sec | ETA 01:46:44
2022-09-07 12:02:11 [INFO] [TRAIN] epoch: 13, iter: 1290/13950, loss: 0.2073, DSC: 81.2893, lr: 0.009164, batch_cost: 0.5060, reader_cost: 0.00035, ips: 47.4323 samples/sec | ETA 01:46:45
2022-09-07 12:02:16 [INFO] [TRAIN] epoch: 13, iter: 1300/13950, loss: 0.2041, DSC: 81.0908, lr: 0.009158, batch_cost: 0.5041, reader_cost: 0.00026, ips: 47.6134 samples/sec | ETA 01:46:16
2022-09-07 12:02:22 [INFO] [TRAIN] epoch: 14, iter: 1310/13950, loss: 0.2179, DSC: 80.0927, lr: 0.009151, batch_cost: 0.6115, reader_cost: 0.12248, ips: 39.2473 samples/sec | ETA 02:08:49
2022-09-07 12:02:27 [INFO] [TRAIN] epoch: 14, iter: 1320/13950, loss: 0.1906, DSC: 82.6380, lr: 0.009145, batch_cost: 0.5058, reader_cost: 0.00029, ips: 47.4513 samples/sec | ETA 01:46:28
2022-09-07 12:02:32 [INFO] [TRAIN] epoch: 14, iter: 1330/13950, loss: 0.1989, DSC: 81.8766, lr: 0.009138, batch_cost: 0.5041, reader_cost: 0.00027, ips: 47.6124 samples/sec | ETA 01:46:01
2022-09-07 12:02:37 [INFO] [TRAIN] epoch: 14, iter: 1340/13950, loss: 0.2205, DSC: 79.6664, lr: 0.009132, batch_cost: 0.5049, reader_cost: 0.00028, ips: 47.5348 samples/sec | ETA 01:46:06
2022-09-07 12:02:42 [INFO] [TRAIN] epoch: 14, iter: 1350/13950, loss: 0.2090, DSC: 80.7240, lr: 0.009125, batch_cost: 0.5056, reader_cost: 0.00027, ips: 47.4644 samples/sec | ETA 01:46:11
2022-09-07 12:02:48 [INFO] [TRAIN] epoch: 14, iter: 1360/13950, loss: 0.2356, DSC: 78.2563, lr: 0.009119, batch_cost: 0.5097, reader_cost: 0.00035, ips: 47.0831 samples/sec | ETA 01:46:57
2022-09-07 12:02:53 [INFO] [TRAIN] epoch: 14, iter: 1370/13950, loss: 0.2003, DSC: 81.4887, lr: 0.009112, batch_cost: 0.5036, reader_cost: 0.00028, ips: 47.6600 samples/sec | ETA 01:45:34
2022-09-07 12:02:58 [INFO] [TRAIN] epoch: 14, iter: 1380/13950, loss: 0.2150, DSC: 80.1202, lr: 0.009106, batch_cost: 0.5048, reader_cost: 0.00028, ips: 47.5417 samples/sec | ETA 01:45:45
2022-09-07 12:03:03 [INFO] [TRAIN] epoch: 14, iter: 1390/13950, loss: 0.1863, DSC: 82.8213, lr: 0.009099, batch_cost: 0.5029, reader_cost: 0.00025, ips: 47.7225 samples/sec | ETA 01:45:16
2022-09-07 12:03:09 [INFO] [TRAIN] epoch: 15, iter: 1400/13950, loss: 0.1879, DSC: 82.8924, lr: 0.009093, batch_cost: 0.6103, reader_cost: 0.12758, ips: 39.3261 samples/sec | ETA 02:07:39
2022-09-07 12:03:14 [INFO] [TRAIN] epoch: 15, iter: 1410/13950, loss: 0.1895, DSC: 82.4572, lr: 0.009086, batch_cost: 0.5030, reader_cost: 0.00027, ips: 47.7137 samples/sec | ETA 01:45:07
2022-09-07 12:03:19 [INFO] [TRAIN] epoch: 15, iter: 1420/13950, loss: 0.1620, DSC: 85.1083, lr: 0.009080, batch_cost: 0.5044, reader_cost: 0.00027, ips: 47.5778 samples/sec | ETA 01:45:20
2022-09-07 12:03:24 [INFO] [TRAIN] epoch: 15, iter: 1430/13950, loss: 0.1834, DSC: 83.0272, lr: 0.009073, batch_cost: 0.5043, reader_cost: 0.00027, ips: 47.5890 samples/sec | ETA 01:45:14
2022-09-07 12:03:29 [INFO] [TRAIN] epoch: 15, iter: 1440/13950, loss: 0.1712, DSC: 84.2611, lr: 0.009067, batch_cost: 0.5046, reader_cost: 0.00028, ips: 47.5620 samples/sec | ETA 01:45:12
2022-09-07 12:03:34 [INFO] [TRAIN] epoch: 15, iter: 1450/13950, loss: 0.1842, DSC: 82.9889, lr: 0.009060, batch_cost: 0.5040, reader_cost: 0.00028, ips: 47.6165 samples/sec | ETA 01:45:00
2022-09-07 12:03:39 [INFO] [TRAIN] epoch: 15, iter: 1460/13950, loss: 0.1967, DSC: 81.8346, lr: 0.009054, batch_cost: 0.5029, reader_cost: 0.00026, ips: 47.7237 samples/sec | ETA 01:44:41
2022-09-07 12:03:44 [INFO] [TRAIN] epoch: 15, iter: 1470/13950, loss: 0.1998, DSC: 81.7178, lr: 0.009047, batch_cost: 0.5052, reader_cost: 0.00033, ips: 47.5103 samples/sec | ETA 01:45:04
2022-09-07 12:03:49 [INFO] [TRAIN] epoch: 15, iter: 1480/13950, loss: 0.1905, DSC: 82.4661, lr: 0.009041, batch_cost: 0.5069, reader_cost: 0.00029, ips: 47.3506 samples/sec | ETA 01:45:20
2022-09-07 12:03:55 [INFO] [TRAIN] epoch: 16, iter: 1490/13950, loss: 0.2219, DSC: 79.1673, lr: 0.009034, batch_cost: 0.6010, reader_cost: 0.11720, ips: 39.9359 samples/sec | ETA 02:04:48
2022-09-07 12:04:00 [INFO] [TRAIN] epoch: 16, iter: 1500/13950, loss: 0.1853, DSC: 83.0110, lr: 0.009027, batch_cost: 0.5062, reader_cost: 0.00028, ips: 47.4152 samples/sec | ETA 01:45:01
2022-09-07 12:04:05 [INFO] [TRAIN] epoch: 16, iter: 1510/13950, loss: 0.1968, DSC: 81.7384, lr: 0.009021, batch_cost: 0.5046, reader_cost: 0.00027, ips: 47.5583 samples/sec | ETA 01:44:37
2022-09-07 12:04:10 [INFO] [TRAIN] epoch: 16, iter: 1520/13950, loss: 0.1881, DSC: 82.5975, lr: 0.009014, batch_cost: 0.5044, reader_cost: 0.00033, ips: 47.5856 samples/sec | ETA 01:44:29
2022-09-07 12:04:15 [INFO] [TRAIN] epoch: 16, iter: 1530/13950, loss: 0.2123, DSC: 80.2369, lr: 0.009008, batch_cost: 0.5053, reader_cost: 0.00035, ips: 47.5007 samples/sec | ETA 01:44:35
2022-09-07 12:04:20 [INFO] [TRAIN] epoch: 16, iter: 1540/13950, loss: 0.1800, DSC: 83.5255, lr: 0.009001, batch_cost: 0.5047, reader_cost: 0.00027, ips: 47.5529 samples/sec | ETA 01:44:23
2022-09-07 12:04:25 [INFO] [TRAIN] epoch: 16, iter: 1550/13950, loss: 0.1715, DSC: 84.2498, lr: 0.008995, batch_cost: 0.5055, reader_cost: 0.00027, ips: 47.4792 samples/sec | ETA 01:44:28
2022-09-07 12:04:30 [INFO] [TRAIN] epoch: 16, iter: 1560/13950, loss: 0.1742, DSC: 83.9684, lr: 0.008988, batch_cost: 0.5052, reader_cost: 0.00034, ips: 47.5102 samples/sec | ETA 01:44:18
2022-09-07 12:04:36 [INFO] [TRAIN] epoch: 16, iter: 1570/13950, loss: 0.1682, DSC: 84.7002, lr: 0.008982, batch_cost: 0.5053, reader_cost: 0.00029, ips: 47.4953 samples/sec | ETA 01:44:15
2022-09-07 12:04:41 [INFO] [TRAIN] epoch: 16, iter: 1580/13950, loss: 0.1785, DSC: 83.3585, lr: 0.008975, batch_cost: 0.5030, reader_cost: 0.00023, ips: 47.7090 samples/sec | ETA 01:43:42
2022-09-07 12:04:47 [INFO] [TRAIN] epoch: 17, iter: 1590/13950, loss: 0.2170, DSC: 79.9662, lr: 0.008969, batch_cost: 0.6041, reader_cost: 0.11398, ips: 39.7271 samples/sec | ETA 02:04:26
2022-09-07 12:04:52 [INFO] [TRAIN] epoch: 17, iter: 1600/13950, loss: 0.1733, DSC: 83.9981, lr: 0.008962, batch_cost: 0.5066, reader_cost: 0.00035, ips: 47.3722 samples/sec | ETA 01:44:16
2022-09-07 12:04:57 [INFO] [TRAIN] epoch: 17, iter: 1610/13950, loss: 0.1635, DSC: 85.1041, lr: 0.008956, batch_cost: 0.5059, reader_cost: 0.00030, ips: 47.4377 samples/sec | ETA 01:44:03
2022-09-07 12:05:02 [INFO] [TRAIN] epoch: 17, iter: 1620/13950, loss: 0.1664, DSC: 84.6847, lr: 0.008949, batch_cost: 0.5036, reader_cost: 0.00027, ips: 47.6593 samples/sec | ETA 01:43:29
2022-09-07 12:05:07 [INFO] [TRAIN] epoch: 17, iter: 1630/13950, loss: 0.1752, DSC: 83.7667, lr: 0.008943, batch_cost: 0.5036, reader_cost: 0.00035, ips: 47.6558 samples/sec | ETA 01:43:24
2022-09-07 12:05:12 [INFO] [TRAIN] epoch: 17, iter: 1640/13950, loss: 0.1686, DSC: 84.5641, lr: 0.008936, batch_cost: 0.5030, reader_cost: 0.00027, ips: 47.7115 samples/sec | ETA 01:43:12
2022-09-07 12:05:17 [INFO] [TRAIN] epoch: 17, iter: 1650/13950, loss: 0.1762, DSC: 83.7494, lr: 0.008930, batch_cost: 0.5035, reader_cost: 0.00027, ips: 47.6634 samples/sec | ETA 01:43:13
2022-09-07 12:05:22 [INFO] [TRAIN] epoch: 17, iter: 1660/13950, loss: 0.1741, DSC: 83.7095, lr: 0.008923, batch_cost: 0.5037, reader_cost: 0.00026, ips: 47.6514 samples/sec | ETA 01:43:09
2022-09-07 12:05:27 [INFO] [TRAIN] epoch: 17, iter: 1670/13950, loss: 0.1798, DSC: 83.3763, lr: 0.008916, batch_cost: 0.5046, reader_cost: 0.00033, ips: 47.5648 samples/sec | ETA 01:43:16
2022-09-07 12:05:33 [INFO] [TRAIN] epoch: 18, iter: 1680/13950, loss: 0.1748, DSC: 83.9362, lr: 0.008910, batch_cost: 0.6277, reader_cost: 0.13410, ips: 38.2321 samples/sec | ETA 02:08:22
2022-09-07 12:05:38 [INFO] [TRAIN] epoch: 18, iter: 1690/13950, loss: 0.1585, DSC: 85.2929, lr: 0.008903, batch_cost: 0.5028, reader_cost: 0.00027, ips: 47.7303 samples/sec | ETA 01:42:44
2022-09-07 12:05:43 [INFO] [TRAIN] epoch: 18, iter: 1700/13950, loss: 0.1591, DSC: 85.5016, lr: 0.008897, batch_cost: 0.5033, reader_cost: 0.00027, ips: 47.6842 samples/sec | ETA 01:42:45
2022-09-07 12:05:48 [INFO] [TRAIN] epoch: 18, iter: 1710/13950, loss: 0.1710, DSC: 84.2760, lr: 0.008890, batch_cost: 0.5035, reader_cost: 0.00027, ips: 47.6675 samples/sec | ETA 01:42:42
2022-09-07 12:05:53 [INFO] [TRAIN] epoch: 18, iter: 1720/13950, loss: 0.1959, DSC: 81.7081, lr: 0.008884, batch_cost: 0.5069, reader_cost: 0.00029, ips: 47.3428 samples/sec | ETA 01:43:19
2022-09-07 12:05:58 [INFO] [TRAIN] epoch: 18, iter: 1730/13950, loss: 0.1781, DSC: 83.6974, lr: 0.008877, batch_cost: 0.5061, reader_cost: 0.00029, ips: 47.4200 samples/sec | ETA 01:43:04
2022-09-07 12:06:03 [INFO] [TRAIN] epoch: 18, iter: 1740/13950, loss: 0.1580, DSC: 85.6699, lr: 0.008871, batch_cost: 0.5044, reader_cost: 0.00027, ips: 47.5857 samples/sec | ETA 01:42:38
2022-09-07 12:06:09 [INFO] [TRAIN] epoch: 18, iter: 1750/13950, loss: 0.1753, DSC: 83.7500, lr: 0.008864, batch_cost: 0.5038, reader_cost: 0.00026, ips: 47.6394 samples/sec | ETA 01:42:26
2022-09-07 12:06:14 [INFO] [TRAIN] epoch: 18, iter: 1760/13950, loss: 0.1595, DSC: 85.3658, lr: 0.008858, batch_cost: 0.5045, reader_cost: 0.00034, ips: 47.5740 samples/sec | ETA 01:42:29
2022-09-07 12:06:20 [INFO] [TRAIN] epoch: 19, iter: 1770/13950, loss: 0.2445, DSC: 76.8091, lr: 0.008851, batch_cost: 0.6108, reader_cost: 0.12892, ips: 39.2956 samples/sec | ETA 02:03:58
2022-09-07 12:06:25 [INFO] [TRAIN] epoch: 19, iter: 1780/13950, loss: 0.1626, DSC: 85.0759, lr: 0.008845, batch_cost: 0.5074, reader_cost: 0.00028, ips: 47.2976 samples/sec | ETA 01:42:55
2022-09-07 12:06:30 [INFO] [TRAIN] epoch: 19, iter: 1790/13950, loss: 0.1555, DSC: 85.7636, lr: 0.008838, batch_cost: 0.5062, reader_cost: 0.00030, ips: 47.4097 samples/sec | ETA 01:42:35
2022-09-07 12:06:35 [INFO] [TRAIN] epoch: 19, iter: 1800/13950, loss: 0.1816, DSC: 82.9081, lr: 0.008831, batch_cost: 0.5059, reader_cost: 0.00034, ips: 47.4385 samples/sec | ETA 01:42:26
2022-09-07 12:06:40 [INFO] [TRAIN] epoch: 19, iter: 1810/13950, loss: 0.1667, DSC: 84.4696, lr: 0.008825, batch_cost: 0.5043, reader_cost: 0.00028, ips: 47.5940 samples/sec | ETA 01:42:01
2022-09-07 12:06:45 [INFO] [TRAIN] epoch: 19, iter: 1820/13950, loss: 0.1569, DSC: 85.5634, lr: 0.008818, batch_cost: 0.5050, reader_cost: 0.00027, ips: 47.5280 samples/sec | ETA 01:42:05
2022-09-07 12:06:50 [INFO] [TRAIN] epoch: 19, iter: 1830/13950, loss: 0.1673, DSC: 84.4710, lr: 0.008812, batch_cost: 0.5043, reader_cost: 0.00028, ips: 47.5874 samples/sec | ETA 01:41:52
2022-09-07 12:06:55 [INFO] [TRAIN] epoch: 19, iter: 1840/13950, loss: 0.1685, DSC: 84.4526, lr: 0.008805, batch_cost: 0.5061, reader_cost: 0.00030, ips: 47.4208 samples/sec | ETA 01:42:08
2022-09-07 12:07:00 [INFO] [TRAIN] epoch: 19, iter: 1850/13950, loss: 0.1583, DSC: 85.4784, lr: 0.008799, batch_cost: 0.5052, reader_cost: 0.00030, ips: 47.5043 samples/sec | ETA 01:41:53
2022-09-07 12:07:05 [INFO] [TRAIN] epoch: 20, iter: 1860/13950, loss: 0.1870, DSC: 82.6687, lr: 0.008792, batch_cost: 0.4754, reader_cost: 0.00024, ips: 50.4859 samples/sec | ETA 01:35:47
2022-09-07 12:07:11 [INFO] [TRAIN] epoch: 20, iter: 1870/13950, loss: 0.1553, DSC: 85.8015, lr: 0.008786, batch_cost: 0.6382, reader_cost: 0.12248, ips: 37.6058 samples/sec | ETA 02:08:29
2022-09-07 12:07:16 [INFO] [TRAIN] epoch: 20, iter: 1880/13950, loss: 0.1566, DSC: 85.6268, lr: 0.008779, batch_cost: 0.5030, reader_cost: 0.00035, ips: 47.7129 samples/sec | ETA 01:41:11
2022-09-07 12:07:21 [INFO] [TRAIN] epoch: 20, iter: 1890/13950, loss: 0.1525, DSC: 85.8346, lr: 0.008773, batch_cost: 0.5026, reader_cost: 0.00027, ips: 47.7538 samples/sec | ETA 01:41:01
2022-09-07 12:07:26 [INFO] [TRAIN] epoch: 20, iter: 1900/13950, loss: 0.1690, DSC: 84.3711, lr: 0.008766, batch_cost: 0.5030, reader_cost: 0.00029, ips: 47.7135 samples/sec | ETA 01:41:01
2022-09-07 12:07:31 [INFO] [TRAIN] epoch: 20, iter: 1910/13950, loss: 0.1567, DSC: 85.3516, lr: 0.008760, batch_cost: 0.5049, reader_cost: 0.00028, ips: 47.5358 samples/sec | ETA 01:41:18
2022-09-07 12:07:36 [INFO] [TRAIN] epoch: 20, iter: 1920/13950, loss: 0.1623, DSC: 85.1140, lr: 0.008753, batch_cost: 0.5066, reader_cost: 0.00028, ips: 47.3717 samples/sec | ETA 01:41:34
2022-09-07 12:07:42 [INFO] [TRAIN] epoch: 20, iter: 1930/13950, loss: 0.1722, DSC: 83.9030, lr: 0.008746, batch_cost: 0.5035, reader_cost: 0.00029, ips: 47.6676 samples/sec | ETA 01:40:51
2022-09-07 12:07:47 [INFO] [TRAIN] epoch: 20, iter: 1940/13950, loss: 0.1584, DSC: 85.3160, lr: 0.008740, batch_cost: 0.5046, reader_cost: 0.00027, ips: 47.5669 samples/sec | ETA 01:40:59
2022-09-07 12:07:52 [INFO] [TRAIN] epoch: 20, iter: 1950/13950, loss: 0.1418, DSC: 87.0691, lr: 0.008733, batch_cost: 0.5041, reader_cost: 0.00025, ips: 47.6091 samples/sec | ETA 01:40:49
2022-09-07 12:07:58 [INFO] [TRAIN] epoch: 21, iter: 1960/13950, loss: 0.1850, DSC: 82.6861, lr: 0.008727, batch_cost: 0.6106, reader_cost: 0.12456, ips: 39.3041 samples/sec | ETA 02:02:01
2022-09-07 12:08:03 [INFO] [TRAIN] epoch: 21, iter: 1970/13950, loss: 0.1535, DSC: 85.6943, lr: 0.008720, batch_cost: 0.5029, reader_cost: 0.00029, ips: 47.7240 samples/sec | ETA 01:40:24
2022-09-07 12:08:08 [INFO] [TRAIN] epoch: 21, iter: 1980/13950, loss: 0.1441, DSC: 86.5824, lr: 0.008714, batch_cost: 0.5038, reader_cost: 0.00028, ips: 47.6357 samples/sec | ETA 01:40:30
2022-09-07 12:08:13 [INFO] [TRAIN] epoch: 21, iter: 1990/13950, loss: 0.1676, DSC: 84.3701, lr: 0.008707, batch_cost: 0.5041, reader_cost: 0.00028, ips: 47.6128 samples/sec | ETA 01:40:28
2022-09-07 12:08:18 [INFO] [TRAIN] epoch: 21, iter: 2000/13950, loss: 0.1421, DSC: 87.0533, lr: 0.008701, batch_cost: 0.5041, reader_cost: 0.00033, ips: 47.6053 samples/sec | ETA 01:40:24
2022-09-07 12:08:18 [INFO] Start evaluating (total_samples: 12, total_iters: 12)...
1/12 [=>............................] - ETA: 21s - batch_cost: 1.9883 - reader cost: 1.1412 2/12 [====>.........................] - ETA: 12s - batch_cost: 1.2618 - reader cost: 0.5730 3/12 [======>.......................] - ETA: 9s - batch_cost: 1.0423 - reader cost: 0.3896 4/12 [=========>....................] - ETA: 9s - batch_cost: 1.1949 - reader cost: 0.4420 5/12 [===========>..................] - ETA: 7s - batch_cost: 1.1269 - reader cost: 0.3580 6/12 [==============>...............] - ETA: 6s - batch_cost: 1.0779 - reader cost: 0.3052 7/12 [================>.............] - ETA: 5s - batch_cost: 1.0097 - reader cost: 0.2644 8/12 [===================>..........] - ETA: 4s - batch_cost: 1.1076 - reader cost: 0.3112 9/12 [=====================>........] - ETA: 3s - batch_cost: 1.0825 - reader cost: 0.279010/12 [========================>.....] - ETA: 2s - batch_cost: 1.0580 - reader cost: 0.253911/12 [==========================>...] - ETA: 1s - batch_cost: 1.0079 - reader cost: 0.232212/12 [==============================] - 12s 972ms/step - batch_cost: 0.9714 - reader cost: 0.2147
2022-09-07 12:08:30 [INFO] [EVAL] #Images: 12, Dice: 0.7935, Loss: 0.227099
2022-09-07 12:08:30 [INFO] [EVAL] Class dice:
[0.9939 0.8587 0.4614 0.8229 0.8268 0.9409 0.6371 0.8615 0.7386]
2022-09-07 12:08:35 [INFO] [EVAL] The model with the best validation mDice (0.7935) was saved at iter 2000.
2022-09-07 12:08:40 [INFO] [TRAIN] epoch: 21, iter: 2010/13950, loss: 0.1554, DSC: 85.8826, lr: 0.008694, batch_cost: 0.5031, reader_cost: 0.00044, ips: 47.7005 samples/sec | ETA 01:40:07
2022-09-07 12:08:45 [INFO] [TRAIN] epoch: 21, iter: 2020/13950, loss: 0.1454, DSC: 86.6078, lr: 0.008687, batch_cost: 0.5037, reader_cost: 0.00027, ips: 47.6489 samples/sec | ETA 01:40:08
2022-09-07 12:08:50 [INFO] [TRAIN] epoch: 21, iter: 2030/13950, loss: 0.1398, DSC: 87.2151, lr: 0.008681, batch_cost: 0.5039, reader_cost: 0.00026, ips: 47.6267 samples/sec | ETA 01:40:06
2022-09-07 12:08:55 [INFO] [TRAIN] epoch: 21, iter: 2040/13950, loss: 0.1585, DSC: 85.3721, lr: 0.008674, batch_cost: 0.5044, reader_cost: 0.00027, ips: 47.5851 samples/sec | ETA 01:40:06
2022-09-07 12:09:02 [INFO] [TRAIN] epoch: 22, iter: 2050/13950, loss: 0.2123, DSC: 79.9810, lr: 0.008668, batch_cost: 0.6289, reader_cost: 0.14031, ips: 38.1635 samples/sec | ETA 02:04:43
2022-09-07 12:09:07 [INFO] [TRAIN] epoch: 22, iter: 2060/13950, loss: 0.1374, DSC: 87.3488, lr: 0.008661, batch_cost: 0.5051, reader_cost: 0.00027, ips: 47.5132 samples/sec | ETA 01:40:05
2022-09-07 12:09:12 [INFO] [TRAIN] epoch: 22, iter: 2070/13950, loss: 0.1566, DSC: 85.4578, lr: 0.008655, batch_cost: 0.5024, reader_cost: 0.00028, ips: 47.7749 samples/sec | ETA 01:39:27
2022-09-07 12:09:17 [INFO] [TRAIN] epoch: 22, iter: 2080/13950, loss: 0.1477, DSC: 86.4653, lr: 0.008648, batch_cost: 0.5042, reader_cost: 0.00040, ips: 47.6041 samples/sec | ETA 01:39:44
2022-09-07 12:09:22 [INFO] [TRAIN] epoch: 22, iter: 2090/13950, loss: 0.1588, DSC: 85.2573, lr: 0.008642, batch_cost: 0.5036, reader_cost: 0.00027, ips: 47.6563 samples/sec | ETA 01:39:32
2022-09-07 12:09:27 [INFO] [TRAIN] epoch: 22, iter: 2100/13950, loss: 0.1537, DSC: 85.8739, lr: 0.008635, batch_cost: 0.5030, reader_cost: 0.00034, ips: 47.7159 samples/sec | ETA 01:39:20
2022-09-07 12:09:32 [INFO] [TRAIN] epoch: 22, iter: 2110/13950, loss: 0.1419, DSC: 86.9741, lr: 0.008628, batch_cost: 0.5039, reader_cost: 0.00027, ips: 47.6306 samples/sec | ETA 01:39:25
2022-09-07 12:09:37 [INFO] [TRAIN] epoch: 22, iter: 2120/13950, loss: 0.1556, DSC: 85.5703, lr: 0.008622, batch_cost: 0.5039, reader_cost: 0.00026, ips: 47.6330 samples/sec | ETA 01:39:20
2022-09-07 12:09:42 [INFO] [TRAIN] epoch: 22, iter: 2130/13950, loss: 0.1572, DSC: 85.2898, lr: 0.008615, batch_cost: 0.5050, reader_cost: 0.00028, ips: 47.5251 samples/sec | ETA 01:39:29
2022-09-07 12:09:48 [INFO] [TRAIN] epoch: 23, iter: 2140/13950, loss: 0.1454, DSC: 86.6636, lr: 0.008609, batch_cost: 0.6017, reader_cost: 0.11939, ips: 39.8856 samples/sec | ETA 01:58:26
2022-09-07 12:09:53 [INFO] [TRAIN] epoch: 23, iter: 2150/13950, loss: 0.1492, DSC: 86.2342, lr: 0.008602, batch_cost: 0.5121, reader_cost: 0.00028, ips: 46.8663 samples/sec | ETA 01:40:42
2022-09-07 12:09:58 [INFO] [TRAIN] epoch: 23, iter: 2160/13950, loss: 0.1572, DSC: 85.4084, lr: 0.008596, batch_cost: 0.5040, reader_cost: 0.00033, ips: 47.6184 samples/sec | ETA 01:39:02
2022-09-07 12:10:03 [INFO] [TRAIN] epoch: 23, iter: 2170/13950, loss: 0.1629, DSC: 85.0747, lr: 0.008589, batch_cost: 0.5053, reader_cost: 0.00029, ips: 47.4982 samples/sec | ETA 01:39:12
2022-09-07 12:10:08 [INFO] [TRAIN] epoch: 23, iter: 2180/13950, loss: 0.1537, DSC: 85.7878, lr: 0.008583, batch_cost: 0.5043, reader_cost: 0.00027, ips: 47.5884 samples/sec | ETA 01:38:55
2022-09-07 12:10:13 [INFO] [TRAIN] epoch: 23, iter: 2190/13950, loss: 0.1529, DSC: 85.7550, lr: 0.008576, batch_cost: 0.5048, reader_cost: 0.00029, ips: 47.5424 samples/sec | ETA 01:38:56
2022-09-07 12:10:18 [INFO] [TRAIN] epoch: 23, iter: 2200/13950, loss: 0.1639, DSC: 84.7797, lr: 0.008569, batch_cost: 0.5032, reader_cost: 0.00033, ips: 47.6991 samples/sec | ETA 01:38:32
2022-09-07 12:10:23 [INFO] [TRAIN] epoch: 23, iter: 2210/13950, loss: 0.1561, DSC: 85.5003, lr: 0.008563, batch_cost: 0.5040, reader_cost: 0.00027, ips: 47.6236 samples/sec | ETA 01:38:36
2022-09-07 12:10:28 [INFO] [TRAIN] epoch: 23, iter: 2220/13950, loss: 0.1607, DSC: 85.1827, lr: 0.008556, batch_cost: 0.5038, reader_cost: 0.00033, ips: 47.6399 samples/sec | ETA 01:38:29
2022-09-07 12:10:33 [INFO] [TRAIN] epoch: 23, iter: 2230/13950, loss: 0.1445, DSC: 86.5443, lr: 0.008550, batch_cost: 0.5032, reader_cost: 0.00024, ips: 47.6922 samples/sec | ETA 01:38:17
2022-09-07 12:10:39 [INFO] [TRAIN] epoch: 24, iter: 2240/13950, loss: 0.1743, DSC: 83.8006, lr: 0.008543, batch_cost: 0.6109, reader_cost: 0.13184, ips: 39.2875 samples/sec | ETA 01:59:13
2022-09-07 12:10:45 [INFO] [TRAIN] epoch: 24, iter: 2250/13950, loss: 0.1541, DSC: 85.7246, lr: 0.008537, batch_cost: 0.5052, reader_cost: 0.00028, ips: 47.5072 samples/sec | ETA 01:38:30
2022-09-07 12:10:50 [INFO] [TRAIN] epoch: 24, iter: 2260/13950, loss: 0.1346, DSC: 87.7316, lr: 0.008530, batch_cost: 0.5021, reader_cost: 0.00033, ips: 47.7967 samples/sec | ETA 01:37:49
2022-09-07 12:10:55 [INFO] [TRAIN] epoch: 24, iter: 2270/13950, loss: 0.1514, DSC: 86.0331, lr: 0.008523, batch_cost: 0.5043, reader_cost: 0.00035, ips: 47.5942 samples/sec | ETA 01:38:09
2022-09-07 12:11:00 [INFO] [TRAIN] epoch: 24, iter: 2280/13950, loss: 0.1485, DSC: 86.1612, lr: 0.008517, batch_cost: 0.5048, reader_cost: 0.00036, ips: 47.5438 samples/sec | ETA 01:38:10
2022-09-07 12:11:05 [INFO] [TRAIN] epoch: 24, iter: 2290/13950, loss: 0.1531, DSC: 85.7852, lr: 0.008510, batch_cost: 0.5074, reader_cost: 0.00035, ips: 47.2988 samples/sec | ETA 01:38:36
2022-09-07 12:11:10 [INFO] [TRAIN] epoch: 24, iter: 2300/13950, loss: 0.1436, DSC: 86.8408, lr: 0.008504, batch_cost: 0.5047, reader_cost: 0.00029, ips: 47.5576 samples/sec | ETA 01:37:59
2022-09-07 12:11:15 [INFO] [TRAIN] epoch: 24, iter: 2310/13950, loss: 0.1424, DSC: 86.8928, lr: 0.008497, batch_cost: 0.5041, reader_cost: 0.00029, ips: 47.6073 samples/sec | ETA 01:37:48
2022-09-07 12:11:20 [INFO] [TRAIN] epoch: 24, iter: 2320/13950, loss: 0.1482, DSC: 86.2103, lr: 0.008491, batch_cost: 0.5041, reader_cost: 0.00025, ips: 47.6103 samples/sec | ETA 01:37:42
2022-09-07 12:11:26 [INFO] [TRAIN] epoch: 25, iter: 2330/13950, loss: 0.1804, DSC: 82.9783, lr: 0.008484, batch_cost: 0.6074, reader_cost: 0.12333, ips: 39.5144 samples/sec | ETA 01:57:37
2022-09-07 12:11:31 [INFO] [TRAIN] epoch: 25, iter: 2340/13950, loss: 0.1502, DSC: 86.1308, lr: 0.008477, batch_cost: 0.5029, reader_cost: 0.00028, ips: 47.7196 samples/sec | ETA 01:37:19
2022-09-07 12:11:36 [INFO] [TRAIN] epoch: 25, iter: 2350/13950, loss: 0.1172, DSC: 89.3383, lr: 0.008471, batch_cost: 0.5037, reader_cost: 0.00035, ips: 47.6437 samples/sec | ETA 01:37:23
2022-09-07 12:11:41 [INFO] [TRAIN] epoch: 25, iter: 2360/13950, loss: 0.1344, DSC: 87.5934, lr: 0.008464, batch_cost: 0.5030, reader_cost: 0.00028, ips: 47.7127 samples/sec | ETA 01:37:09
2022-09-07 12:11:46 [INFO] [TRAIN] epoch: 25, iter: 2370/13950, loss: 0.1666, DSC: 84.4388, lr: 0.008458, batch_cost: 0.5053, reader_cost: 0.00028, ips: 47.4922 samples/sec | ETA 01:37:31
2022-09-07 12:11:51 [INFO] [TRAIN] epoch: 25, iter: 2380/13950, loss: 0.1531, DSC: 85.7306, lr: 0.008451, batch_cost: 0.5033, reader_cost: 0.00027, ips: 47.6877 samples/sec | ETA 01:37:02
2022-09-07 12:11:56 [INFO] [TRAIN] epoch: 25, iter: 2390/13950, loss: 0.1448, DSC: 86.4483, lr: 0.008445, batch_cost: 0.5048, reader_cost: 0.00029, ips: 47.5449 samples/sec | ETA 01:37:15
2022-09-07 12:12:01 [INFO] [TRAIN] epoch: 25, iter: 2400/13950, loss: 0.1431, DSC: 86.9469, lr: 0.008438, batch_cost: 0.5048, reader_cost: 0.00029, ips: 47.5404 samples/sec | ETA 01:37:10
2022-09-07 12:12:06 [INFO] [TRAIN] epoch: 25, iter: 2410/13950, loss: 0.1566, DSC: 85.4696, lr: 0.008431, batch_cost: 0.5063, reader_cost: 0.00036, ips: 47.3993 samples/sec | ETA 01:37:23
2022-09-07 12:12:12 [INFO] [TRAIN] epoch: 26, iter: 2420/13950, loss: 0.1667, DSC: 84.6373, lr: 0.008425, batch_cost: 0.6171, reader_cost: 0.13350, ips: 38.8927 samples/sec | ETA 01:58:34
2022-09-07 12:12:18 [INFO] [TRAIN] epoch: 26, iter: 2430/13950, loss: 0.1388, DSC: 87.3085, lr: 0.008418, batch_cost: 0.5161, reader_cost: 0.00029, ips: 46.5065 samples/sec | ETA 01:39:04
2022-09-07 12:12:23 [INFO] [TRAIN] epoch: 26, iter: 2440/13950, loss: 0.1438, DSC: 86.5831, lr: 0.008412, batch_cost: 0.5036, reader_cost: 0.00029, ips: 47.6547 samples/sec | ETA 01:36:36
2022-09-07 12:12:28 [INFO] [TRAIN] epoch: 26, iter: 2450/13950, loss: 0.1337, DSC: 87.6924, lr: 0.008405, batch_cost: 0.5044, reader_cost: 0.00041, ips: 47.5845 samples/sec | ETA 01:36:40
2022-09-07 12:12:33 [INFO] [TRAIN] epoch: 26, iter: 2460/13950, loss: 0.1523, DSC: 85.9706, lr: 0.008399, batch_cost: 0.5048, reader_cost: 0.00029, ips: 47.5464 samples/sec | ETA 01:36:39
2022-09-07 12:12:38 [INFO] [TRAIN] epoch: 26, iter: 2470/13950, loss: 0.1457, DSC: 86.5148, lr: 0.008392, batch_cost: 0.5032, reader_cost: 0.00027, ips: 47.6962 samples/sec | ETA 01:36:16
2022-09-07 12:12:43 [INFO] [TRAIN] epoch: 26, iter: 2480/13950, loss: 0.1308, DSC: 87.9530, lr: 0.008385, batch_cost: 0.5039, reader_cost: 0.00033, ips: 47.6323 samples/sec | ETA 01:36:19
2022-09-07 12:12:48 [INFO] [TRAIN] epoch: 26, iter: 2490/13950, loss: 0.1420, DSC: 86.8987, lr: 0.008379, batch_cost: 0.5103, reader_cost: 0.00028, ips: 47.0294 samples/sec | ETA 01:37:28
2022-09-07 12:12:53 [INFO] [TRAIN] epoch: 26, iter: 2500/13950, loss: 0.1517, DSC: 85.8258, lr: 0.008372, batch_cost: 0.5030, reader_cost: 0.00028, ips: 47.7095 samples/sec | ETA 01:35:59
2022-09-07 12:12:58 [INFO] [TRAIN] epoch: 26, iter: 2510/13950, loss: 0.1851, DSC: 82.6325, lr: 0.008366, batch_cost: 0.5028, reader_cost: 0.00024, ips: 47.7370 samples/sec | ETA 01:35:51
2022-09-07 12:13:04 [INFO] [TRAIN] epoch: 27, iter: 2520/13950, loss: 0.1922, DSC: 81.9012, lr: 0.008359, batch_cost: 0.6121, reader_cost: 0.12760, ips: 39.2068 samples/sec | ETA 01:56:36
2022-09-07 12:13:09 [INFO] [TRAIN] epoch: 27, iter: 2530/13950, loss: 0.1710, DSC: 83.9677, lr: 0.008353, batch_cost: 0.5075, reader_cost: 0.00029, ips: 47.2933 samples/sec | ETA 01:36:35
2022-09-07 12:13:14 [INFO] [TRAIN] epoch: 27, iter: 2540/13950, loss: 0.1431, DSC: 86.8020, lr: 0.008346, batch_cost: 0.5067, reader_cost: 0.00031, ips: 47.3652 samples/sec | ETA 01:36:21
2022-09-07 12:13:19 [INFO] [TRAIN] epoch: 27, iter: 2550/13950, loss: 0.1410, DSC: 86.9284, lr: 0.008339, batch_cost: 0.5057, reader_cost: 0.00029, ips: 47.4594 samples/sec | ETA 01:36:04
2022-09-07 12:13:24 [INFO] [TRAIN] epoch: 27, iter: 2560/13950, loss: 0.1421, DSC: 86.8264, lr: 0.008333, batch_cost: 0.5048, reader_cost: 0.00027, ips: 47.5479 samples/sec | ETA 01:35:49
2022-09-07 12:13:29 [INFO] [TRAIN] epoch: 27, iter: 2570/13950, loss: 0.1404, DSC: 87.1344, lr: 0.008326, batch_cost: 0.5050, reader_cost: 0.00027, ips: 47.5218 samples/sec | ETA 01:35:47
2022-09-07 12:13:34 [INFO] [TRAIN] epoch: 27, iter: 2580/13950, loss: 0.1689, DSC: 83.9349, lr: 0.008320, batch_cost: 0.5047, reader_cost: 0.00028, ips: 47.5492 samples/sec | ETA 01:35:38
2022-09-07 12:13:39 [INFO] [TRAIN] epoch: 27, iter: 2590/13950, loss: 0.1452, DSC: 86.6265, lr: 0.008313, batch_cost: 0.5038, reader_cost: 0.00027, ips: 47.6337 samples/sec | ETA 01:35:23
2022-09-07 12:13:45 [INFO] [TRAIN] epoch: 27, iter: 2600/13950, loss: 0.1494, DSC: 86.1689, lr: 0.008306, batch_cost: 0.5044, reader_cost: 0.00033, ips: 47.5860 samples/sec | ETA 01:35:24
2022-09-07 12:13:51 [INFO] [TRAIN] epoch: 28, iter: 2610/13950, loss: 0.1786, DSC: 83.2164, lr: 0.008300, batch_cost: 0.6641, reader_cost: 0.13183, ips: 36.1377 samples/sec | ETA 02:05:31
2022-09-07 12:13:56 [INFO] [TRAIN] epoch: 28, iter: 2620/13950, loss: 0.1501, DSC: 86.1046, lr: 0.008293, batch_cost: 0.5064, reader_cost: 0.00028, ips: 47.3941 samples/sec | ETA 01:35:37
2022-09-07 12:14:01 [INFO] [TRAIN] epoch: 28, iter: 2630/13950, loss: 0.1283, DSC: 88.2573, lr: 0.008287, batch_cost: 0.5050, reader_cost: 0.00034, ips: 47.5254 samples/sec | ETA 01:35:16
2022-09-07 12:14:06 [INFO] [TRAIN] epoch: 28, iter: 2640/13950, loss: 0.1390, DSC: 87.1317, lr: 0.008280, batch_cost: 0.5060, reader_cost: 0.00034, ips: 47.4270 samples/sec | ETA 01:35:23
2022-09-07 12:14:11 [INFO] [TRAIN] epoch: 28, iter: 2650/13950, loss: 0.1289, DSC: 88.0000, lr: 0.008273, batch_cost: 0.5051, reader_cost: 0.00028, ips: 47.5119 samples/sec | ETA 01:35:08
2022-09-07 12:14:16 [INFO] [TRAIN] epoch: 28, iter: 2660/13950, loss: 0.1381, DSC: 87.2978, lr: 0.008267, batch_cost: 0.5054, reader_cost: 0.00027, ips: 47.4882 samples/sec | ETA 01:35:05
2022-09-07 12:14:21 [INFO] [TRAIN] epoch: 28, iter: 2670/13950, loss: 0.1388, DSC: 87.2264, lr: 0.008260, batch_cost: 0.5043, reader_cost: 0.00028, ips: 47.5878 samples/sec | ETA 01:34:48
2022-09-07 12:14:27 [INFO] [TRAIN] epoch: 28, iter: 2680/13950, loss: 0.1391, DSC: 87.1488, lr: 0.008254, batch_cost: 0.5054, reader_cost: 0.00027, ips: 47.4904 samples/sec | ETA 01:34:55
2022-09-07 12:14:32 [INFO] [TRAIN] epoch: 28, iter: 2690/13950, loss: 0.1251, DSC: 88.4782, lr: 0.008247, batch_cost: 0.5051, reader_cost: 0.00028, ips: 47.5196 samples/sec | ETA 01:34:46
2022-09-07 12:14:38 [INFO] [TRAIN] epoch: 29, iter: 2700/13950, loss: 0.1533, DSC: 85.6732, lr: 0.008241, batch_cost: 0.6210, reader_cost: 0.13655, ips: 38.6497 samples/sec | ETA 01:56:25
2022-09-07 12:14:43 [INFO] [TRAIN] epoch: 29, iter: 2710/13950, loss: 0.1502, DSC: 85.9448, lr: 0.008234, batch_cost: 0.5061, reader_cost: 0.00028, ips: 47.4194 samples/sec | ETA 01:34:48
2022-09-07 12:14:48 [INFO] [TRAIN] epoch: 29, iter: 2720/13950, loss: 0.1595, DSC: 85.1229, lr: 0.008227, batch_cost: 0.5041, reader_cost: 0.00027, ips: 47.6127 samples/sec | ETA 01:34:20
2022-09-07 12:14:53 [INFO] [TRAIN] epoch: 29, iter: 2730/13950, loss: 0.1436, DSC: 86.8274, lr: 0.008221, batch_cost: 0.5090, reader_cost: 0.00029, ips: 47.1512 samples/sec | ETA 01:35:10
2022-09-07 12:14:58 [INFO] [TRAIN] epoch: 29, iter: 2740/13950, loss: 0.1484, DSC: 86.2855, lr: 0.008214, batch_cost: 0.5063, reader_cost: 0.00029, ips: 47.4045 samples/sec | ETA 01:34:35
2022-09-07 12:15:03 [INFO] [TRAIN] epoch: 29, iter: 2750/13950, loss: 0.1303, DSC: 88.0207, lr: 0.008208, batch_cost: 0.5028, reader_cost: 0.00029, ips: 47.7281 samples/sec | ETA 01:33:51
2022-09-07 12:15:08 [INFO] [TRAIN] epoch: 29, iter: 2760/13950, loss: 0.1403, DSC: 87.0503, lr: 0.008201, batch_cost: 0.5022, reader_cost: 0.00027, ips: 47.7878 samples/sec | ETA 01:33:39
2022-09-07 12:15:13 [INFO] [TRAIN] epoch: 29, iter: 2770/13950, loss: 0.1419, DSC: 86.9279, lr: 0.008194, batch_cost: 0.5035, reader_cost: 0.00035, ips: 47.6661 samples/sec | ETA 01:33:49
2022-09-07 12:15:18 [INFO] [TRAIN] epoch: 29, iter: 2780/13950, loss: 0.1198, DSC: 88.9465, lr: 0.008188, batch_cost: 0.5030, reader_cost: 0.00028, ips: 47.7174 samples/sec | ETA 01:33:38
2022-09-07 12:15:23 [INFO] [TRAIN] epoch: 30, iter: 2790/13950, loss: 0.1820, DSC: 82.8579, lr: 0.008181, batch_cost: 0.4745, reader_cost: 0.00023, ips: 50.5813 samples/sec | ETA 01:28:15
2022-09-07 12:15:29 [INFO] [TRAIN] epoch: 30, iter: 2800/13950, loss: 0.1496, DSC: 86.1379, lr: 0.008175, batch_cost: 0.6558, reader_cost: 0.12881, ips: 36.5949 samples/sec | ETA 02:01:52
2022-09-07 12:15:35 [INFO] [TRAIN] epoch: 30, iter: 2810/13950, loss: 0.1358, DSC: 87.5042, lr: 0.008168, batch_cost: 0.5031, reader_cost: 0.00027, ips: 47.7051 samples/sec | ETA 01:33:24
2022-09-07 12:15:40 [INFO] [TRAIN] epoch: 30, iter: 2820/13950, loss: 0.1305, DSC: 88.0295, lr: 0.008161, batch_cost: 0.5027, reader_cost: 0.00026, ips: 47.7407 samples/sec | ETA 01:33:15
2022-09-07 12:15:45 [INFO] [TRAIN] epoch: 30, iter: 2830/13950, loss: 0.1480, DSC: 85.9663, lr: 0.008155, batch_cost: 0.5040, reader_cost: 0.00028, ips: 47.6180 samples/sec | ETA 01:33:24
2022-09-07 12:15:50 [INFO] [TRAIN] epoch: 30, iter: 2840/13950, loss: 0.1281, DSC: 88.3727, lr: 0.008148, batch_cost: 0.5042, reader_cost: 0.00028, ips: 47.6006 samples/sec | ETA 01:33:21
2022-09-07 12:15:55 [INFO] [TRAIN] epoch: 30, iter: 2850/13950, loss: 0.1353, DSC: 87.4632, lr: 0.008142, batch_cost: 0.5065, reader_cost: 0.00029, ips: 47.3795 samples/sec | ETA 01:33:42
2022-09-07 12:16:00 [INFO] [TRAIN] epoch: 30, iter: 2860/13950, loss: 0.1394, DSC: 87.0343, lr: 0.008135, batch_cost: 0.5049, reader_cost: 0.00027, ips: 47.5335 samples/sec | ETA 01:33:19
2022-09-07 12:16:05 [INFO] [TRAIN] epoch: 30, iter: 2870/13950, loss: 0.1190, DSC: 89.1528, lr: 0.008128, batch_cost: 0.5037, reader_cost: 0.00034, ips: 47.6507 samples/sec | ETA 01:33:00
2022-09-07 12:16:10 [INFO] [TRAIN] epoch: 30, iter: 2880/13950, loss: 0.1559, DSC: 85.3720, lr: 0.008122, batch_cost: 0.5031, reader_cost: 0.00024, ips: 47.7058 samples/sec | ETA 01:32:49
2022-09-07 12:16:16 [INFO] [TRAIN] epoch: 31, iter: 2890/13950, loss: 0.1875, DSC: 82.2724, lr: 0.008115, batch_cost: 0.6141, reader_cost: 0.12073, ips: 39.0803 samples/sec | ETA 01:53:12
2022-09-07 12:16:21 [INFO] [TRAIN] epoch: 31, iter: 2900/13950, loss: 0.1282, DSC: 88.0527, lr: 0.008109, batch_cost: 0.5011, reader_cost: 0.00027, ips: 47.8904 samples/sec | ETA 01:32:17
2022-09-07 12:16:26 [INFO] [TRAIN] epoch: 31, iter: 2910/13950, loss: 0.1499, DSC: 86.1183, lr: 0.008102, batch_cost: 0.5031, reader_cost: 0.00034, ips: 47.7032 samples/sec | ETA 01:32:34
2022-09-07 12:16:31 [INFO] [TRAIN] epoch: 31, iter: 2920/13950, loss: 0.1434, DSC: 86.6635, lr: 0.008095, batch_cost: 0.5035, reader_cost: 0.00026, ips: 47.6668 samples/sec | ETA 01:32:33
2022-09-07 12:16:36 [INFO] [TRAIN] epoch: 31, iter: 2930/13950, loss: 0.1385, DSC: 87.1268, lr: 0.008089, batch_cost: 0.5044, reader_cost: 0.00029, ips: 47.5839 samples/sec | ETA 01:32:38
2022-09-07 12:16:41 [INFO] [TRAIN] epoch: 31, iter: 2940/13950, loss: 0.1290, DSC: 88.1863, lr: 0.008082, batch_cost: 0.5043, reader_cost: 0.00027, ips: 47.5916 samples/sec | ETA 01:32:32
2022-09-07 12:16:46 [INFO] [TRAIN] epoch: 31, iter: 2950/13950, loss: 0.1375, DSC: 87.3392, lr: 0.008076, batch_cost: 0.5053, reader_cost: 0.00036, ips: 47.4978 samples/sec | ETA 01:32:38
2022-09-07 12:16:51 [INFO] [TRAIN] epoch: 31, iter: 2960/13950, loss: 0.1430, DSC: 86.5295, lr: 0.008069, batch_cost: 0.5042, reader_cost: 0.00027, ips: 47.5991 samples/sec | ETA 01:32:21
2022-09-07 12:16:56 [INFO] [TRAIN] epoch: 31, iter: 2970/13950, loss: 0.1370, DSC: 87.3413, lr: 0.008062, batch_cost: 0.5065, reader_cost: 0.00029, ips: 47.3813 samples/sec | ETA 01:32:41
2022-09-07 12:17:03 [INFO] [TRAIN] epoch: 32, iter: 2980/13950, loss: 0.1282, DSC: 88.1284, lr: 0.008056, batch_cost: 0.6521, reader_cost: 0.16062, ips: 36.8016 samples/sec | ETA 01:59:14
2022-09-07 12:17:08 [INFO] [TRAIN] epoch: 32, iter: 2990/13950, loss: 0.1461, DSC: 86.2977, lr: 0.008049, batch_cost: 0.5085, reader_cost: 0.00030, ips: 47.1988 samples/sec | ETA 01:32:53
2022-09-07 12:17:13 [INFO] [TRAIN] epoch: 32, iter: 3000/13950, loss: 0.1296, DSC: 88.0542, lr: 0.008043, batch_cost: 0.5065, reader_cost: 0.00031, ips: 47.3841 samples/sec | ETA 01:32:26
2022-09-07 12:17:13 [INFO] Start evaluating (total_samples: 12, total_iters: 12)...
1/12 [=>............................] - ETA: 22s - batch_cost: 2.0035 - reader cost: 1.1571 2/12 [====>.........................] - ETA: 12s - batch_cost: 1.2708 - reader cost: 0.5814 3/12 [======>.......................] - ETA: 9s - batch_cost: 1.0485 - reader cost: 0.3955 4/12 [=========>....................] - ETA: 9s - batch_cost: 1.2042 - reader cost: 0.4514 5/12 [===========>..................] - ETA: 7s - batch_cost: 1.1344 - reader cost: 0.3654 6/12 [==============>...............] - ETA: 6s - batch_cost: 1.0872 - reader cost: 0.3143 7/12 [================>.............] - ETA: 5s - batch_cost: 1.0175 - reader cost: 0.2722 8/12 [===================>..........] - ETA: 4s - batch_cost: 1.1194 - reader cost: 0.3230 9/12 [=====================>........] - ETA: 3s - batch_cost: 1.0925 - reader cost: 0.289410/12 [========================>.....] - ETA: 2s - batch_cost: 1.0670 - reader cost: 0.263511/12 [==========================>...] - ETA: 1s - batch_cost: 1.0160 - reader cost: 0.241012/12 [==============================] - 12s 979ms/step - batch_cost: 0.9787 - reader cost: 0.2227
2022-09-07 12:17:25 [INFO] [EVAL] #Images: 12, Dice: 0.8123, Loss: 0.207095
2022-09-07 12:17:25 [INFO] [EVAL] Class dice:
[0.9942 0.8936 0.5192 0.8764 0.8409 0.9417 0.6097 0.8726 0.7625]
2022-09-07 12:17:30 [INFO] [EVAL] The model with the best validation mDice (0.8123) was saved at iter 3000.
2022-09-07 12:17:35 [INFO] [TRAIN] epoch: 32, iter: 3010/13950, loss: 0.1403, DSC: 87.0535, lr: 0.008036, batch_cost: 0.5053, reader_cost: 0.00053, ips: 47.4971 samples/sec | ETA 01:32:07
2022-09-07 12:17:40 [INFO] [TRAIN] epoch: 32, iter: 3020/13950, loss: 0.1387, DSC: 87.1131, lr: 0.008029, batch_cost: 0.5047, reader_cost: 0.00040, ips: 47.5573 samples/sec | ETA 01:31:55
2022-09-07 12:17:45 [INFO] [TRAIN] epoch: 32, iter: 3030/13950, loss: 0.1529, DSC: 85.7226, lr: 0.008023, batch_cost: 0.5039, reader_cost: 0.00025, ips: 47.6278 samples/sec | ETA 01:31:42
2022-09-07 12:17:51 [INFO] [TRAIN] epoch: 32, iter: 3040/13950, loss: 0.1397, DSC: 86.9959, lr: 0.008016, batch_cost: 0.5048, reader_cost: 0.00033, ips: 47.5391 samples/sec | ETA 01:31:47
2022-09-07 12:17:56 [INFO] [TRAIN] epoch: 32, iter: 3050/13950, loss: 0.1339, DSC: 87.6137, lr: 0.008009, batch_cost: 0.5063, reader_cost: 0.00043, ips: 47.3981 samples/sec | ETA 01:31:59
2022-09-07 12:18:01 [INFO] [TRAIN] epoch: 32, iter: 3060/13950, loss: 0.1381, DSC: 87.2564, lr: 0.008003, batch_cost: 0.5069, reader_cost: 0.00027, ips: 47.3423 samples/sec | ETA 01:32:00
2022-09-07 12:18:07 [INFO] [TRAIN] epoch: 33, iter: 3070/13950, loss: 0.1955, DSC: 81.5692, lr: 0.007996, batch_cost: 0.6047, reader_cost: 0.12382, ips: 39.6894 samples/sec | ETA 01:49:39
2022-09-07 12:18:12 [INFO] [TRAIN] epoch: 33, iter: 3080/13950, loss: 0.1578, DSC: 85.2546, lr: 0.007990, batch_cost: 0.5074, reader_cost: 0.00030, ips: 47.3017 samples/sec | ETA 01:31:55
2022-09-07 12:18:17 [INFO] [TRAIN] epoch: 33, iter: 3090/13950, loss: 0.1488, DSC: 86.2819, lr: 0.007983, batch_cost: 0.5035, reader_cost: 0.00028, ips: 47.6628 samples/sec | ETA 01:31:08
2022-09-07 12:18:22 [INFO] [TRAIN] epoch: 33, iter: 3100/13950, loss: 0.1397, DSC: 87.2511, lr: 0.007976, batch_cost: 0.5037, reader_cost: 0.00027, ips: 47.6464 samples/sec | ETA 01:31:05
2022-09-07 12:18:27 [INFO] [TRAIN] epoch: 33, iter: 3110/13950, loss: 0.1363, DSC: 87.3665, lr: 0.007970, batch_cost: 0.5032, reader_cost: 0.00027, ips: 47.6984 samples/sec | ETA 01:30:54
2022-09-07 12:18:32 [INFO] [TRAIN] epoch: 33, iter: 3120/13950, loss: 0.1288, DSC: 88.1687, lr: 0.007963, batch_cost: 0.5037, reader_cost: 0.00026, ips: 47.6481 samples/sec | ETA 01:30:54
2022-09-07 12:18:37 [INFO] [TRAIN] epoch: 33, iter: 3130/13950, loss: 0.1412, DSC: 86.8485, lr: 0.007957, batch_cost: 0.5067, reader_cost: 0.00029, ips: 47.3639 samples/sec | ETA 01:31:22
2022-09-07 12:18:42 [INFO] [TRAIN] epoch: 33, iter: 3140/13950, loss: 0.1285, DSC: 88.1268, lr: 0.007950, batch_cost: 0.5035, reader_cost: 0.00033, ips: 47.6695 samples/sec | ETA 01:30:42
2022-09-07 12:18:47 [INFO] [TRAIN] epoch: 33, iter: 3150/13950, loss: 0.1246, DSC: 88.5695, lr: 0.007943, batch_cost: 0.5039, reader_cost: 0.00027, ips: 47.6313 samples/sec | ETA 01:30:41
2022-09-07 12:18:52 [INFO] [TRAIN] epoch: 33, iter: 3160/13950, loss: 0.1658, DSC: 84.3633, lr: 0.007937, batch_cost: 0.5041, reader_cost: 0.00024, ips: 47.6055 samples/sec | ETA 01:30:39
2022-09-07 12:18:59 [INFO] [TRAIN] epoch: 34, iter: 3170/13950, loss: 0.1453, DSC: 86.6210, lr: 0.007930, batch_cost: 0.6125, reader_cost: 0.12720, ips: 39.1823 samples/sec | ETA 01:50:02
2022-09-07 12:19:04 [INFO] [TRAIN] epoch: 34, iter: 3180/13950, loss: 0.1534, DSC: 85.7002, lr: 0.007923, batch_cost: 0.5062, reader_cost: 0.00028, ips: 47.4110 samples/sec | ETA 01:30:51
2022-09-07 12:19:09 [INFO] [TRAIN] epoch: 34, iter: 3190/13950, loss: 0.1489, DSC: 86.1758, lr: 0.007917, batch_cost: 0.5031, reader_cost: 0.00027, ips: 47.7061 samples/sec | ETA 01:30:13
2022-09-07 12:19:14 [INFO] [TRAIN] epoch: 34, iter: 3200/13950, loss: 0.1332, DSC: 87.6341, lr: 0.007910, batch_cost: 0.5040, reader_cost: 0.00027, ips: 47.6217 samples/sec | ETA 01:30:17
2022-09-07 12:19:19 [INFO] [TRAIN] epoch: 34, iter: 3210/13950, loss: 0.1419, DSC: 86.8114, lr: 0.007904, batch_cost: 0.5029, reader_cost: 0.00026, ips: 47.7264 samples/sec | ETA 01:30:00
2022-09-07 12:19:24 [INFO] [TRAIN] epoch: 34, iter: 3220/13950, loss: 0.1433, DSC: 86.5978, lr: 0.007897, batch_cost: 0.5046, reader_cost: 0.00028, ips: 47.5650 samples/sec | ETA 01:30:14
2022-09-07 12:19:29 [INFO] [TRAIN] epoch: 34, iter: 3230/13950, loss: 0.1364, DSC: 87.2912, lr: 0.007890, batch_cost: 0.5041, reader_cost: 0.00026, ips: 47.6069 samples/sec | ETA 01:30:04
2022-09-07 12:19:34 [INFO] [TRAIN] epoch: 34, iter: 3240/13950, loss: 0.1151, DSC: 89.4121, lr: 0.007884, batch_cost: 0.5038, reader_cost: 0.00034, ips: 47.6357 samples/sec | ETA 01:29:55
2022-09-07 12:19:39 [INFO] [TRAIN] epoch: 34, iter: 3250/13950, loss: 0.1371, DSC: 87.2941, lr: 0.007877, batch_cost: 0.5043, reader_cost: 0.00026, ips: 47.5895 samples/sec | ETA 01:29:56
2022-09-07 12:19:45 [INFO] [TRAIN] epoch: 35, iter: 3260/13950, loss: 0.1522, DSC: 85.8338, lr: 0.007870, batch_cost: 0.6003, reader_cost: 0.11724, ips: 39.9824 samples/sec | ETA 01:46:56
2022-09-07 12:19:50 [INFO] [TRAIN] epoch: 35, iter: 3270/13950, loss: 0.1292, DSC: 88.0563, lr: 0.007864, batch_cost: 0.5041, reader_cost: 0.00027, ips: 47.6108 samples/sec | ETA 01:29:43
2022-09-07 12:19:55 [INFO] [TRAIN] epoch: 35, iter: 3280/13950, loss: 0.1416, DSC: 86.9864, lr: 0.007857, batch_cost: 0.5060, reader_cost: 0.00030, ips: 47.4327 samples/sec | ETA 01:29:58
2022-09-07 12:20:00 [INFO] [TRAIN] epoch: 35, iter: 3290/13950, loss: 0.1247, DSC: 88.4195, lr: 0.007851, batch_cost: 0.5039, reader_cost: 0.00026, ips: 47.6266 samples/sec | ETA 01:29:31
2022-09-07 12:20:05 [INFO] [TRAIN] epoch: 35, iter: 3300/13950, loss: 0.1286, DSC: 88.1092, lr: 0.007844, batch_cost: 0.5045, reader_cost: 0.00028, ips: 47.5702 samples/sec | ETA 01:29:33
2022-09-07 12:20:10 [INFO] [TRAIN] epoch: 35, iter: 3310/13950, loss: 0.1352, DSC: 87.6128, lr: 0.007837, batch_cost: 0.5008, reader_cost: 0.00027, ips: 47.9218 samples/sec | ETA 01:28:48
2022-09-07 12:20:15 [INFO] [TRAIN] epoch: 35, iter: 3320/13950, loss: 0.1358, DSC: 87.2320, lr: 0.007831, batch_cost: 0.5022, reader_cost: 0.00027, ips: 47.7930 samples/sec | ETA 01:28:58
2022-09-07 12:20:20 [INFO] [TRAIN] epoch: 35, iter: 3330/13950, loss: 0.1438, DSC: 86.4121, lr: 0.007824, batch_cost: 0.5021, reader_cost: 0.00027, ips: 47.8012 samples/sec | ETA 01:28:52
2022-09-07 12:20:25 [INFO] [TRAIN] epoch: 35, iter: 3340/13950, loss: 0.1440, DSC: 86.5960, lr: 0.007817, batch_cost: 0.5041, reader_cost: 0.00029, ips: 47.6065 samples/sec | ETA 01:29:08
2022-09-07 12:20:31 [INFO] [TRAIN] epoch: 36, iter: 3350/13950, loss: 0.1536, DSC: 86.0051, lr: 0.007811, batch_cost: 0.5922, reader_cost: 0.11379, ips: 40.5290 samples/sec | ETA 01:44:36
2022-09-07 12:20:36 [INFO] [TRAIN] epoch: 36, iter: 3360/13950, loss: 0.1259, DSC: 88.6372, lr: 0.007804, batch_cost: 0.5109, reader_cost: 0.00027, ips: 46.9778 samples/sec | ETA 01:30:10
2022-09-07 12:20:41 [INFO] [TRAIN] epoch: 36, iter: 3370/13950, loss: 0.1266, DSC: 88.2761, lr: 0.007798, batch_cost: 0.5046, reader_cost: 0.00034, ips: 47.5642 samples/sec | ETA 01:28:58
2022-09-07 12:20:46 [INFO] [TRAIN] epoch: 36, iter: 3380/13950, loss: 0.1284, DSC: 88.2826, lr: 0.007791, batch_cost: 0.5056, reader_cost: 0.00034, ips: 47.4683 samples/sec | ETA 01:29:04
2022-09-07 12:20:51 [INFO] [TRAIN] epoch: 36, iter: 3390/13950, loss: 0.1422, DSC: 86.7621, lr: 0.007784, batch_cost: 0.5041, reader_cost: 0.00028, ips: 47.6053 samples/sec | ETA 01:28:43
2022-09-07 12:20:56 [INFO] [TRAIN] epoch: 36, iter: 3400/13950, loss: 0.1284, DSC: 88.0442, lr: 0.007778, batch_cost: 0.5056, reader_cost: 0.00029, ips: 47.4672 samples/sec | ETA 01:28:54
2022-09-07 12:21:01 [INFO] [TRAIN] epoch: 36, iter: 3410/13950, loss: 0.1662, DSC: 84.2350, lr: 0.007771, batch_cost: 0.5051, reader_cost: 0.00028, ips: 47.5134 samples/sec | ETA 01:28:43
2022-09-07 12:21:06 [INFO] [TRAIN] epoch: 36, iter: 3420/13950, loss: 0.1310, DSC: 87.7421, lr: 0.007764, batch_cost: 0.5056, reader_cost: 0.00028, ips: 47.4694 samples/sec | ETA 01:28:43
2022-09-07 12:21:12 [INFO] [TRAIN] epoch: 36, iter: 3430/13950, loss: 0.1407, DSC: 86.8432, lr: 0.007758, batch_cost: 0.5050, reader_cost: 0.00032, ips: 47.5231 samples/sec | ETA 01:28:32
2022-09-07 12:21:17 [INFO] [TRAIN] epoch: 36, iter: 3440/13950, loss: 0.1328, DSC: 87.6284, lr: 0.007751, batch_cost: 0.5051, reader_cost: 0.00025, ips: 47.5189 samples/sec | ETA 01:28:28
2022-09-07 12:21:23 [INFO] [TRAIN] epoch: 37, iter: 3450/13950, loss: 0.1743, DSC: 83.4346, lr: 0.007744, batch_cost: 0.5931, reader_cost: 0.10734, ips: 40.4634 samples/sec | ETA 01:43:47
2022-09-07 12:21:28 [INFO] [TRAIN] epoch: 37, iter: 3460/13950, loss: 0.1567, DSC: 85.4130, lr: 0.007738, batch_cost: 0.5043, reader_cost: 0.00028, ips: 47.5924 samples/sec | ETA 01:28:09
2022-09-07 12:21:33 [INFO] [TRAIN] epoch: 37, iter: 3470/13950, loss: 0.1304, DSC: 88.0447, lr: 0.007731, batch_cost: 0.5024, reader_cost: 0.00034, ips: 47.7731 samples/sec | ETA 01:27:44
2022-09-07 12:21:38 [INFO] [TRAIN] epoch: 37, iter: 3480/13950, loss: 0.1389, DSC: 87.2156, lr: 0.007725, batch_cost: 0.5019, reader_cost: 0.00028, ips: 47.8154 samples/sec | ETA 01:27:35
2022-09-07 12:21:43 [INFO] [TRAIN] epoch: 37, iter: 3490/13950, loss: 0.1471, DSC: 86.4487, lr: 0.007718, batch_cost: 0.5022, reader_cost: 0.00034, ips: 47.7884 samples/sec | ETA 01:27:33
2022-09-07 12:21:48 [INFO] [TRAIN] epoch: 37, iter: 3500/13950, loss: 0.1239, DSC: 88.6734, lr: 0.007711, batch_cost: 0.5036, reader_cost: 0.00028, ips: 47.6592 samples/sec | ETA 01:27:42
2022-09-07 12:21:53 [INFO] [TRAIN] epoch: 37, iter: 3510/13950, loss: 0.1259, DSC: 88.3997, lr: 0.007705, batch_cost: 0.5036, reader_cost: 0.00036, ips: 47.6549 samples/sec | ETA 01:27:37
2022-09-07 12:21:58 [INFO] [TRAIN] epoch: 37, iter: 3520/13950, loss: 0.1402, DSC: 86.8814, lr: 0.007698, batch_cost: 0.5074, reader_cost: 0.00030, ips: 47.2966 samples/sec | ETA 01:28:12
2022-09-07 12:22:03 [INFO] [TRAIN] epoch: 37, iter: 3530/13950, loss: 0.1504, DSC: 85.8354, lr: 0.007691, batch_cost: 0.5037, reader_cost: 0.00026, ips: 47.6484 samples/sec | ETA 01:27:28
2022-09-07 12:22:09 [INFO] [TRAIN] epoch: 38, iter: 3540/13950, loss: 0.1420, DSC: 86.8827, lr: 0.007685, batch_cost: 0.6258, reader_cost: 0.12851, ips: 38.3524 samples/sec | ETA 01:48:34
2022-09-07 12:22:14 [INFO] [TRAIN] epoch: 38, iter: 3550/13950, loss: 0.1201, DSC: 88.8715, lr: 0.007678, batch_cost: 0.5025, reader_cost: 0.00030, ips: 47.7651 samples/sec | ETA 01:27:05
2022-09-07 12:22:19 [INFO] [TRAIN] epoch: 38, iter: 3560/13950, loss: 0.1289, DSC: 88.1819, lr: 0.007671, batch_cost: 0.5035, reader_cost: 0.00028, ips: 47.6703 samples/sec | ETA 01:27:10
2022-09-07 12:22:24 [INFO] [TRAIN] epoch: 38, iter: 3570/13950, loss: 0.1316, DSC: 87.8257, lr: 0.007665, batch_cost: 0.5042, reader_cost: 0.00027, ips: 47.6039 samples/sec | ETA 01:27:13
2022-09-07 12:22:29 [INFO] [TRAIN] epoch: 38, iter: 3580/13950, loss: 0.1390, DSC: 87.0657, lr: 0.007658, batch_cost: 0.5053, reader_cost: 0.00027, ips: 47.4957 samples/sec | ETA 01:27:20
2022-09-07 12:22:34 [INFO] [TRAIN] epoch: 38, iter: 3590/13950, loss: 0.1122, DSC: 89.7861, lr: 0.007651, batch_cost: 0.5045, reader_cost: 0.00027, ips: 47.5678 samples/sec | ETA 01:27:07
2022-09-07 12:22:39 [INFO] [TRAIN] epoch: 38, iter: 3600/13950, loss: 0.1260, DSC: 88.3074, lr: 0.007645, batch_cost: 0.5040, reader_cost: 0.00028, ips: 47.6181 samples/sec | ETA 01:26:56
2022-09-07 12:22:44 [INFO] [TRAIN] epoch: 38, iter: 3610/13950, loss: 0.1284, DSC: 87.9939, lr: 0.007638, batch_cost: 0.5050, reader_cost: 0.00030, ips: 47.5239 samples/sec | ETA 01:27:01
2022-09-07 12:22:49 [INFO] [TRAIN] epoch: 38, iter: 3620/13950, loss: 0.1402, DSC: 86.9184, lr: 0.007632, batch_cost: 0.5051, reader_cost: 0.00035, ips: 47.5118 samples/sec | ETA 01:26:58
2022-09-07 12:22:55 [INFO] [TRAIN] epoch: 39, iter: 3630/13950, loss: 0.1650, DSC: 84.4849, lr: 0.007625, batch_cost: 0.6056, reader_cost: 0.11891, ips: 39.6282 samples/sec | ETA 01:44:10
2022-09-07 12:23:01 [INFO] [TRAIN] epoch: 39, iter: 3640/13950, loss: 0.1335, DSC: 87.5376, lr: 0.007618, batch_cost: 0.5062, reader_cost: 0.00028, ips: 47.4101 samples/sec | ETA 01:26:59
2022-09-07 12:23:06 [INFO] [TRAIN] epoch: 39, iter: 3650/13950, loss: 0.1316, DSC: 88.0625, lr: 0.007612, batch_cost: 0.5042, reader_cost: 0.00036, ips: 47.6032 samples/sec | ETA 01:26:32
2022-09-07 12:23:11 [INFO] [TRAIN] epoch: 39, iter: 3660/13950, loss: 0.1263, DSC: 88.4754, lr: 0.007605, batch_cost: 0.5045, reader_cost: 0.00028, ips: 47.5748 samples/sec | ETA 01:26:30
2022-09-07 12:23:16 [INFO] [TRAIN] epoch: 39, iter: 3670/13950, loss: 0.1491, DSC: 86.1324, lr: 0.007598, batch_cost: 0.5049, reader_cost: 0.00028, ips: 47.5364 samples/sec | ETA 01:26:30
2022-09-07 12:23:21 [INFO] [TRAIN] epoch: 39, iter: 3680/13950, loss: 0.1284, DSC: 88.0699, lr: 0.007592, batch_cost: 0.5080, reader_cost: 0.00027, ips: 47.2434 samples/sec | ETA 01:26:57
2022-09-07 12:23:26 [INFO] [TRAIN] epoch: 39, iter: 3690/13950, loss: 0.1147, DSC: 89.3730, lr: 0.007585, batch_cost: 0.5052, reader_cost: 0.00028, ips: 47.5030 samples/sec | ETA 01:26:23
2022-09-07 12:23:31 [INFO] [TRAIN] epoch: 39, iter: 3700/13950, loss: 0.1310, DSC: 87.8991, lr: 0.007578, batch_cost: 0.5047, reader_cost: 0.00027, ips: 47.5494 samples/sec | ETA 01:26:13
2022-09-07 12:23:36 [INFO] [TRAIN] epoch: 39, iter: 3710/13950, loss: 0.1445, DSC: 86.2567, lr: 0.007572, batch_cost: 0.5045, reader_cost: 0.00028, ips: 47.5731 samples/sec | ETA 01:26:05
2022-09-07 12:23:41 [INFO] [TRAIN] epoch: 40, iter: 3720/13950, loss: 0.2152, DSC: 79.4081, lr: 0.007565, batch_cost: 0.4745, reader_cost: 0.00022, ips: 50.5769 samples/sec | ETA 01:20:54
2022-09-07 12:23:47 [INFO] [TRAIN] epoch: 40, iter: 3730/13950, loss: 0.1345, DSC: 87.5014, lr: 0.007558, batch_cost: 0.6219, reader_cost: 0.11020, ips: 38.5928 samples/sec | ETA 01:45:55
2022-09-07 12:23:52 [INFO] [TRAIN] epoch: 40, iter: 3740/13950, loss: 0.1199, DSC: 88.6988, lr: 0.007552, batch_cost: 0.5052, reader_cost: 0.00028, ips: 47.5099 samples/sec | ETA 01:25:57
2022-09-07 12:23:57 [INFO] [TRAIN] epoch: 40, iter: 3750/13950, loss: 0.1338, DSC: 87.5482, lr: 0.007545, batch_cost: 0.5030, reader_cost: 0.00029, ips: 47.7133 samples/sec | ETA 01:25:30
2022-09-07 12:24:02 [INFO] [TRAIN] epoch: 40, iter: 3760/13950, loss: 0.1433, DSC: 86.5885, lr: 0.007538, batch_cost: 0.5027, reader_cost: 0.00027, ips: 47.7436 samples/sec | ETA 01:25:22
2022-09-07 12:24:07 [INFO] [TRAIN] epoch: 40, iter: 3770/13950, loss: 0.1200, DSC: 88.8413, lr: 0.007532, batch_cost: 0.5044, reader_cost: 0.00028, ips: 47.5827 samples/sec | ETA 01:25:34
2022-09-07 12:24:12 [INFO] [TRAIN] epoch: 40, iter: 3780/13950, loss: 0.1367, DSC: 87.2067, lr: 0.007525, batch_cost: 0.5044, reader_cost: 0.00028, ips: 47.5838 samples/sec | ETA 01:25:29
2022-09-07 12:24:17 [INFO] [TRAIN] epoch: 40, iter: 3790/13950, loss: 0.1178, DSC: 89.2767, lr: 0.007518, batch_cost: 0.5044, reader_cost: 0.00028, ips: 47.5790 samples/sec | ETA 01:25:24
2022-09-07 12:24:22 [INFO] [TRAIN] epoch: 40, iter: 3800/13950, loss: 0.1212, DSC: 88.8948, lr: 0.007512, batch_cost: 0.5049, reader_cost: 0.00028, ips: 47.5329 samples/sec | ETA 01:25:24
2022-09-07 12:24:27 [INFO] [TRAIN] epoch: 40, iter: 3810/13950, loss: 0.1272, DSC: 88.3099, lr: 0.007505, batch_cost: 0.5032, reader_cost: 0.00025, ips: 47.6922 samples/sec | ETA 01:25:02
2022-09-07 12:24:33 [INFO] [TRAIN] epoch: 41, iter: 3820/13950, loss: 0.2038, DSC: 80.4990, lr: 0.007498, batch_cost: 0.6063, reader_cost: 0.10685, ips: 39.5819 samples/sec | ETA 01:42:22
2022-09-07 12:24:38 [INFO] [TRAIN] epoch: 41, iter: 3830/13950, loss: 0.1395, DSC: 86.8363, lr: 0.007492, batch_cost: 0.5036, reader_cost: 0.00027, ips: 47.6569 samples/sec | ETA 01:24:56
2022-09-07 12:24:43 [INFO] [TRAIN] epoch: 41, iter: 3840/13950, loss: 0.1301, DSC: 87.8598, lr: 0.007485, batch_cost: 0.5032, reader_cost: 0.00027, ips: 47.6951 samples/sec | ETA 01:24:47
2022-09-07 12:24:48 [INFO] [TRAIN] epoch: 41, iter: 3850/13950, loss: 0.1195, DSC: 89.0183, lr: 0.007478, batch_cost: 0.5035, reader_cost: 0.00027, ips: 47.6663 samples/sec | ETA 01:24:45
2022-09-07 12:24:53 [INFO] [TRAIN] epoch: 41, iter: 3860/13950, loss: 0.1357, DSC: 87.3055, lr: 0.007472, batch_cost: 0.5043, reader_cost: 0.00027, ips: 47.5880 samples/sec | ETA 01:24:48
2022-09-07 12:24:58 [INFO] [TRAIN] epoch: 41, iter: 3870/13950, loss: 0.1133, DSC: 89.5216, lr: 0.007465, batch_cost: 0.5053, reader_cost: 0.00029, ips: 47.4961 samples/sec | ETA 01:24:53
2022-09-07 12:25:03 [INFO] [TRAIN] epoch: 41, iter: 3880/13950, loss: 0.1252, DSC: 88.3457, lr: 0.007458, batch_cost: 0.5044, reader_cost: 0.00028, ips: 47.5834 samples/sec | ETA 01:24:39
2022-09-07 12:25:09 [INFO] [TRAIN] epoch: 41, iter: 3890/13950, loss: 0.1152, DSC: 89.3941, lr: 0.007452, batch_cost: 0.5037, reader_cost: 0.00027, ips: 47.6472 samples/sec | ETA 01:24:27
2022-09-07 12:25:14 [INFO] [TRAIN] epoch: 41, iter: 3900/13950, loss: 0.1422, DSC: 86.7845, lr: 0.007445, batch_cost: 0.5046, reader_cost: 0.00035, ips: 47.5643 samples/sec | ETA 01:24:31
2022-09-07 12:25:20 [INFO] [TRAIN] epoch: 42, iter: 3910/13950, loss: 0.1530, DSC: 85.7339, lr: 0.007438, batch_cost: 0.6058, reader_cost: 0.12122, ips: 39.6180 samples/sec | ETA 01:41:22
2022-09-07 12:25:25 [INFO] [TRAIN] epoch: 42, iter: 3920/13950, loss: 0.1288, DSC: 88.3802, lr: 0.007432, batch_cost: 0.5047, reader_cost: 0.00034, ips: 47.5489 samples/sec | ETA 01:24:22
2022-09-07 12:25:30 [INFO] [TRAIN] epoch: 42, iter: 3930/13950, loss: 0.1426, DSC: 86.8197, lr: 0.007425, batch_cost: 0.5045, reader_cost: 0.00027, ips: 47.5676 samples/sec | ETA 01:24:15
2022-09-07 12:25:35 [INFO] [TRAIN] epoch: 42, iter: 3940/13950, loss: 0.1273, DSC: 88.2469, lr: 0.007418, batch_cost: 0.5048, reader_cost: 0.00029, ips: 47.5414 samples/sec | ETA 01:24:13
2022-09-07 12:25:40 [INFO] [TRAIN] epoch: 42, iter: 3950/13950, loss: 0.1384, DSC: 87.1660, lr: 0.007412, batch_cost: 0.5043, reader_cost: 0.00026, ips: 47.5871 samples/sec | ETA 01:24:03
2022-09-07 12:25:45 [INFO] [TRAIN] epoch: 42, iter: 3960/13950, loss: 0.1429, DSC: 86.6883, lr: 0.007405, batch_cost: 0.5035, reader_cost: 0.00025, ips: 47.6691 samples/sec | ETA 01:23:49
2022-09-07 12:25:50 [INFO] [TRAIN] epoch: 42, iter: 3970/13950, loss: 0.1293, DSC: 87.9553, lr: 0.007398, batch_cost: 0.5027, reader_cost: 0.00031, ips: 47.7393 samples/sec | ETA 01:23:37
2022-09-07 12:25:55 [INFO] [TRAIN] epoch: 42, iter: 3980/13950, loss: 0.1442, DSC: 86.5218, lr: 0.007392, batch_cost: 0.5050, reader_cost: 0.00028, ips: 47.5226 samples/sec | ETA 01:23:55
2022-09-07 12:26:00 [INFO] [TRAIN] epoch: 42, iter: 3990/13950, loss: 0.1158, DSC: 89.2511, lr: 0.007385, batch_cost: 0.5034, reader_cost: 0.00033, ips: 47.6736 samples/sec | ETA 01:23:34
2022-09-07 12:26:06 [INFO] [TRAIN] epoch: 43, iter: 4000/13950, loss: 0.1246, DSC: 88.3765, lr: 0.007378, batch_cost: 0.5826, reader_cost: 0.10494, ips: 41.1919 samples/sec | ETA 01:36:37
2022-09-07 12:26:06 [INFO] Start evaluating (total_samples: 12, total_iters: 12)...
1/12 [=>............................] - ETA: 23s - batch_cost: 2.1063 - reader cost: 1.2598 2/12 [====>.........................] - ETA: 13s - batch_cost: 1.3213 - reader cost: 0.6326 3/12 [======>.......................] - ETA: 9s - batch_cost: 1.0817 - reader cost: 0.4293 4/12 [=========>....................] - ETA: 9s - batch_cost: 1.2235 - reader cost: 0.4711 5/12 [===========>..................] - ETA: 8s - batch_cost: 1.1498 - reader cost: 0.3811 6/12 [==============>...............] - ETA: 6s - batch_cost: 1.0970 - reader cost: 0.3247 7/12 [================>.............] - ETA: 5s - batch_cost: 1.0258 - reader cost: 0.2811 8/12 [===================>..........] - ETA: 4s - batch_cost: 1.1218 - reader cost: 0.3257 9/12 [=====================>........] - ETA: 3s - batch_cost: 1.0949 - reader cost: 0.291910/12 [========================>.....] - ETA: 2s - batch_cost: 1.0688 - reader cost: 0.265811/12 [==========================>...] - ETA: 1s - batch_cost: 1.0178 - reader cost: 0.243212/12 [==============================] - 12s 981ms/step - batch_cost: 0.9805 - reader cost: 0.2247
2022-09-07 12:26:18 [INFO] [EVAL] #Images: 12, Dice: 0.8107, Loss: 0.210965
2022-09-07 12:26:18 [INFO] [EVAL] Class dice:
[0.9948 0.8865 0.5197 0.8683 0.8489 0.9411 0.6215 0.8716 0.7436]
2022-09-07 12:26:20 [INFO] [EVAL] The model with the best validation mDice (0.8123) was saved at iter 3000.
2022-09-07 12:26:25 [INFO] [TRAIN] epoch: 43, iter: 4010/13950, loss: 0.1461, DSC: 86.3217, lr: 0.007372, batch_cost: 0.5049, reader_cost: 0.00043, ips: 47.5350 samples/sec | ETA 01:23:38
2022-09-07 12:26:30 [INFO] [TRAIN] epoch: 43, iter: 4020/13950, loss: 0.1243, DSC: 88.4145, lr: 0.007365, batch_cost: 0.5053, reader_cost: 0.00026, ips: 47.4981 samples/sec | ETA 01:23:37
2022-09-07 12:26:35 [INFO] [TRAIN] epoch: 43, iter: 4030/13950, loss: 0.1239, DSC: 88.5173, lr: 0.007358, batch_cost: 0.5047, reader_cost: 0.00031, ips: 47.5540 samples/sec | ETA 01:23:26
2022-09-07 12:26:40 [INFO] [TRAIN] epoch: 43, iter: 4040/13950, loss: 0.1511, DSC: 85.7445, lr: 0.007352, batch_cost: 0.5048, reader_cost: 0.00026, ips: 47.5456 samples/sec | ETA 01:23:22
2022-09-07 12:26:45 [INFO] [TRAIN] epoch: 43, iter: 4050/13950, loss: 0.1269, DSC: 88.1912, lr: 0.007345, batch_cost: 0.5040, reader_cost: 0.00026, ips: 47.6188 samples/sec | ETA 01:23:09
2022-09-07 12:26:50 [INFO] [TRAIN] epoch: 43, iter: 4060/13950, loss: 0.1094, DSC: 90.0750, lr: 0.007338, batch_cost: 0.5002, reader_cost: 0.00024, ips: 47.9820 samples/sec | ETA 01:22:26
2022-09-07 12:26:55 [INFO] [TRAIN] epoch: 43, iter: 4070/13950, loss: 0.1176, DSC: 89.1939, lr: 0.007332, batch_cost: 0.5059, reader_cost: 0.00029, ips: 47.4432 samples/sec | ETA 01:23:17
2022-09-07 12:27:00 [INFO] [TRAIN] epoch: 43, iter: 4080/13950, loss: 0.1148, DSC: 89.3949, lr: 0.007325, batch_cost: 0.5057, reader_cost: 0.00029, ips: 47.4628 samples/sec | ETA 01:23:10
2022-09-07 12:27:05 [INFO] [TRAIN] epoch: 43, iter: 4090/13950, loss: 0.1373, DSC: 87.2374, lr: 0.007318, batch_cost: 0.5035, reader_cost: 0.00024, ips: 47.6618 samples/sec | ETA 01:22:44
2022-09-07 12:27:11 [INFO] [TRAIN] epoch: 44, iter: 4100/13950, loss: 0.1959, DSC: 81.1754, lr: 0.007312, batch_cost: 0.6066, reader_cost: 0.10794, ips: 39.5635 samples/sec | ETA 01:39:35
2022-09-07 12:27:16 [INFO] [TRAIN] epoch: 44, iter: 4110/13950, loss: 0.1170, DSC: 89.3320, lr: 0.007305, batch_cost: 0.5060, reader_cost: 0.00029, ips: 47.4275 samples/sec | ETA 01:22:59
2022-09-07 12:27:22 [INFO] [TRAIN] epoch: 44, iter: 4120/13950, loss: 0.1394, DSC: 86.8942, lr: 0.007298, batch_cost: 0.5062, reader_cost: 0.00029, ips: 47.4145 samples/sec | ETA 01:22:55
2022-09-07 12:27:27 [INFO] [TRAIN] epoch: 44, iter: 4130/13950, loss: 0.1136, DSC: 89.5773, lr: 0.007292, batch_cost: 0.5068, reader_cost: 0.00029, ips: 47.3602 samples/sec | ETA 01:22:56
2022-09-07 12:27:32 [INFO] [TRAIN] epoch: 44, iter: 4140/13950, loss: 0.1664, DSC: 84.1648, lr: 0.007285, batch_cost: 0.5046, reader_cost: 0.00029, ips: 47.5593 samples/sec | ETA 01:22:30
2022-09-07 12:27:37 [INFO] [TRAIN] epoch: 44, iter: 4150/13950, loss: 0.1332, DSC: 87.5292, lr: 0.007278, batch_cost: 0.5041, reader_cost: 0.00028, ips: 47.6060 samples/sec | ETA 01:22:20
2022-09-07 12:27:42 [INFO] [TRAIN] epoch: 44, iter: 4160/13950, loss: 0.1211, DSC: 88.9002, lr: 0.007272, batch_cost: 0.5039, reader_cost: 0.00028, ips: 47.6251 samples/sec | ETA 01:22:13
2022-09-07 12:27:47 [INFO] [TRAIN] epoch: 44, iter: 4170/13950, loss: 0.1131, DSC: 89.5980, lr: 0.007265, batch_cost: 0.5047, reader_cost: 0.00030, ips: 47.5484 samples/sec | ETA 01:22:16
2022-09-07 12:27:52 [INFO] [TRAIN] epoch: 44, iter: 4180/13950, loss: 0.1186, DSC: 89.0463, lr: 0.007258, batch_cost: 0.5048, reader_cost: 0.00026, ips: 47.5412 samples/sec | ETA 01:22:12
2022-09-07 12:27:58 [INFO] [TRAIN] epoch: 45, iter: 4190/13950, loss: 0.1221, DSC: 88.6422, lr: 0.007252, batch_cost: 0.6176, reader_cost: 0.12917, ips: 38.8584 samples/sec | ETA 01:40:28
2022-09-07 12:28:03 [INFO] [TRAIN] epoch: 45, iter: 4200/13950, loss: 0.1113, DSC: 89.6305, lr: 0.007245, batch_cost: 0.5076, reader_cost: 0.00043, ips: 47.2782 samples/sec | ETA 01:22:29
2022-09-07 12:28:08 [INFO] [TRAIN] epoch: 45, iter: 4210/13950, loss: 0.1243, DSC: 88.4573, lr: 0.007238, batch_cost: 0.5036, reader_cost: 0.00027, ips: 47.6547 samples/sec | ETA 01:21:45
2022-09-07 12:28:13 [INFO] [TRAIN] epoch: 45, iter: 4220/13950, loss: 0.1311, DSC: 87.8782, lr: 0.007231, batch_cost: 0.5041, reader_cost: 0.00028, ips: 47.6050 samples/sec | ETA 01:21:45
2022-09-07 12:28:18 [INFO] [TRAIN] epoch: 45, iter: 4230/13950, loss: 0.1300, DSC: 87.9472, lr: 0.007225, batch_cost: 0.5050, reader_cost: 0.00027, ips: 47.5268 samples/sec | ETA 01:21:48
2022-09-07 12:28:23 [INFO] [TRAIN] epoch: 45, iter: 4240/13950, loss: 0.1099, DSC: 89.9218, lr: 0.007218, batch_cost: 0.5042, reader_cost: 0.00027, ips: 47.6019 samples/sec | ETA 01:21:35
2022-09-07 12:28:28 [INFO] [TRAIN] epoch: 45, iter: 4250/13950, loss: 0.1141, DSC: 89.4043, lr: 0.007211, batch_cost: 0.5058, reader_cost: 0.00034, ips: 47.4489 samples/sec | ETA 01:21:46
2022-09-07 12:28:33 [INFO] [TRAIN] epoch: 45, iter: 4260/13950, loss: 0.1210, DSC: 88.8627, lr: 0.007205, batch_cost: 0.5042, reader_cost: 0.00029, ips: 47.5964 samples/sec | ETA 01:21:26
2022-09-07 12:28:38 [INFO] [TRAIN] epoch: 45, iter: 4270/13950, loss: 0.1107, DSC: 89.6437, lr: 0.007198, batch_cost: 0.5045, reader_cost: 0.00031, ips: 47.5728 samples/sec | ETA 01:21:23
2022-09-07 12:28:44 [INFO] [TRAIN] epoch: 46, iter: 4280/13950, loss: 0.1673, DSC: 84.1623, lr: 0.007191, batch_cost: 0.5958, reader_cost: 0.11203, ips: 40.2850 samples/sec | ETA 01:36:00
2022-09-07 12:28:49 [INFO] [TRAIN] epoch: 46, iter: 4290/13950, loss: 0.1215, DSC: 88.7615, lr: 0.007185, batch_cost: 0.5071, reader_cost: 0.00034, ips: 47.3282 samples/sec | ETA 01:21:38
2022-09-07 12:28:54 [INFO] [TRAIN] epoch: 46, iter: 4300/13950, loss: 0.1591, DSC: 84.9578, lr: 0.007178, batch_cost: 0.5054, reader_cost: 0.00028, ips: 47.4886 samples/sec | ETA 01:21:16
2022-09-07 12:29:00 [INFO] [TRAIN] epoch: 46, iter: 4310/13950, loss: 0.1199, DSC: 89.0403, lr: 0.007171, batch_cost: 0.5026, reader_cost: 0.00027, ips: 47.7470 samples/sec | ETA 01:20:45
2022-09-07 12:29:05 [INFO] [TRAIN] epoch: 46, iter: 4320/13950, loss: 0.1032, DSC: 90.5272, lr: 0.007165, batch_cost: 0.5029, reader_cost: 0.00027, ips: 47.7252 samples/sec | ETA 01:20:42
2022-09-07 12:29:10 [INFO] [TRAIN] epoch: 46, iter: 4330/13950, loss: 0.1448, DSC: 86.4272, lr: 0.007158, batch_cost: 0.5035, reader_cost: 0.00027, ips: 47.6672 samples/sec | ETA 01:20:43
2022-09-07 12:29:15 [INFO] [TRAIN] epoch: 46, iter: 4340/13950, loss: 0.1417, DSC: 86.7031, lr: 0.007151, batch_cost: 0.5047, reader_cost: 0.00028, ips: 47.5523 samples/sec | ETA 01:20:50
2022-09-07 12:29:20 [INFO] [TRAIN] epoch: 46, iter: 4350/13950, loss: 0.1243, DSC: 88.4877, lr: 0.007144, batch_cost: 0.5043, reader_cost: 0.00028, ips: 47.5921 samples/sec | ETA 01:20:41
2022-09-07 12:29:25 [INFO] [TRAIN] epoch: 46, iter: 4360/13950, loss: 0.1214, DSC: 88.7326, lr: 0.007138, batch_cost: 0.5054, reader_cost: 0.00033, ips: 47.4841 samples/sec | ETA 01:20:47
2022-09-07 12:29:30 [INFO] [TRAIN] epoch: 46, iter: 4370/13950, loss: 0.1310, DSC: 87.7526, lr: 0.007131, batch_cost: 0.5010, reader_cost: 0.00020, ips: 47.9068 samples/sec | ETA 01:19:59
2022-09-07 12:29:36 [INFO] [TRAIN] epoch: 47, iter: 4380/13950, loss: 0.1883, DSC: 81.9245, lr: 0.007124, batch_cost: 0.6059, reader_cost: 0.12166, ips: 39.6081 samples/sec | ETA 01:36:38
2022-09-07 12:29:41 [INFO] [TRAIN] epoch: 47, iter: 4390/13950, loss: 0.1149, DSC: 89.3426, lr: 0.007118, batch_cost: 0.5043, reader_cost: 0.00025, ips: 47.5940 samples/sec | ETA 01:20:20
2022-09-07 12:29:46 [INFO] [TRAIN] epoch: 47, iter: 4400/13950, loss: 0.1360, DSC: 87.2650, lr: 0.007111, batch_cost: 0.5051, reader_cost: 0.00027, ips: 47.5198 samples/sec | ETA 01:20:23
2022-09-07 12:29:51 [INFO] [TRAIN] epoch: 47, iter: 4410/13950, loss: 0.1110, DSC: 89.8332, lr: 0.007104, batch_cost: 0.5050, reader_cost: 0.00026, ips: 47.5260 samples/sec | ETA 01:20:17
2022-09-07 12:29:56 [INFO] [TRAIN] epoch: 47, iter: 4420/13950, loss: 0.1240, DSC: 88.5654, lr: 0.007098, batch_cost: 0.5044, reader_cost: 0.00027, ips: 47.5845 samples/sec | ETA 01:20:06
2022-09-07 12:30:01 [INFO] [TRAIN] epoch: 47, iter: 4430/13950, loss: 0.1126, DSC: 89.5667, lr: 0.007091, batch_cost: 0.5040, reader_cost: 0.00027, ips: 47.6177 samples/sec | ETA 01:19:58
2022-09-07 12:30:06 [INFO] [TRAIN] epoch: 47, iter: 4440/13950, loss: 0.1162, DSC: 89.0970, lr: 0.007084, batch_cost: 0.5052, reader_cost: 0.00026, ips: 47.5086 samples/sec | ETA 01:20:04
2022-09-07 12:30:11 [INFO] [TRAIN] epoch: 47, iter: 4450/13950, loss: 0.1368, DSC: 87.0963, lr: 0.007077, batch_cost: 0.5068, reader_cost: 0.00028, ips: 47.3520 samples/sec | ETA 01:20:15
2022-09-07 12:30:16 [INFO] [TRAIN] epoch: 47, iter: 4460/13950, loss: 0.1258, DSC: 88.4272, lr: 0.007071, batch_cost: 0.5045, reader_cost: 0.00025, ips: 47.5715 samples/sec | ETA 01:19:47
2022-09-07 12:30:22 [INFO] [TRAIN] epoch: 48, iter: 4470/13950, loss: 0.1635, DSC: 84.5438, lr: 0.007064, batch_cost: 0.5910, reader_cost: 0.10472, ips: 40.6058 samples/sec | ETA 01:33:23
2022-09-07 12:30:27 [INFO] [TRAIN] epoch: 48, iter: 4480/13950, loss: 0.1332, DSC: 87.6298, lr: 0.007057, batch_cost: 0.5045, reader_cost: 0.00027, ips: 47.5733 samples/sec | ETA 01:19:37
2022-09-07 12:30:32 [INFO] [TRAIN] epoch: 48, iter: 4490/13950, loss: 0.1215, DSC: 88.7341, lr: 0.007051, batch_cost: 0.5029, reader_cost: 0.00025, ips: 47.7190 samples/sec | ETA 01:19:17
2022-09-07 12:30:37 [INFO] [TRAIN] epoch: 48, iter: 4500/13950, loss: 0.1229, DSC: 88.6565, lr: 0.007044, batch_cost: 0.5032, reader_cost: 0.00025, ips: 47.6909 samples/sec | ETA 01:19:15
2022-09-07 12:30:42 [INFO] [TRAIN] epoch: 48, iter: 4510/13950, loss: 0.1161, DSC: 89.3767, lr: 0.007037, batch_cost: 0.5041, reader_cost: 0.00026, ips: 47.6080 samples/sec | ETA 01:19:18
2022-09-07 12:30:47 [INFO] [TRAIN] epoch: 48, iter: 4520/13950, loss: 0.1193, DSC: 88.8772, lr: 0.007030, batch_cost: 0.5061, reader_cost: 0.00029, ips: 47.4186 samples/sec | ETA 01:19:32
2022-09-07 12:30:52 [INFO] [TRAIN] epoch: 48, iter: 4530/13950, loss: 0.1346, DSC: 87.2583, lr: 0.007024, batch_cost: 0.5044, reader_cost: 0.00026, ips: 47.5835 samples/sec | ETA 01:19:11
2022-09-07 12:30:57 [INFO] [TRAIN] epoch: 48, iter: 4540/13950, loss: 0.1342, DSC: 87.3509, lr: 0.007017, batch_cost: 0.5050, reader_cost: 0.00028, ips: 47.5259 samples/sec | ETA 01:19:11
2022-09-07 12:31:02 [INFO] [TRAIN] epoch: 48, iter: 4550/13950, loss: 0.1347, DSC: 87.3839, lr: 0.007010, batch_cost: 0.5050, reader_cost: 0.00026, ips: 47.5234 samples/sec | ETA 01:19:07
2022-09-07 12:31:08 [INFO] [TRAIN] epoch: 49, iter: 4560/13950, loss: 0.2043, DSC: 80.4261, lr: 0.007004, batch_cost: 0.5840, reader_cost: 0.10032, ips: 41.0945 samples/sec | ETA 01:31:23
2022-09-07 12:31:13 [INFO] [TRAIN] epoch: 49, iter: 4570/13950, loss: 0.1252, DSC: 88.4057, lr: 0.006997, batch_cost: 0.5071, reader_cost: 0.00027, ips: 47.3312 samples/sec | ETA 01:19:16
2022-09-07 12:31:18 [INFO] [TRAIN] epoch: 49, iter: 4580/13950, loss: 0.1184, DSC: 88.9744, lr: 0.006990, batch_cost: 0.5024, reader_cost: 0.00026, ips: 47.7736 samples/sec | ETA 01:18:27
2022-09-07 12:31:23 [INFO] [TRAIN] epoch: 49, iter: 4590/13950, loss: 0.1095, DSC: 89.9615, lr: 0.006984, batch_cost: 0.5005, reader_cost: 0.00026, ips: 47.9496 samples/sec | ETA 01:18:04
2022-09-07 12:31:28 [INFO] [TRAIN] epoch: 49, iter: 4600/13950, loss: 0.1372, DSC: 87.2129, lr: 0.006977, batch_cost: 0.5024, reader_cost: 0.00027, ips: 47.7679 samples/sec | ETA 01:18:17
2022-09-07 12:31:33 [INFO] [TRAIN] epoch: 49, iter: 4610/13950, loss: 0.1361, DSC: 87.3437, lr: 0.006970, batch_cost: 0.5037, reader_cost: 0.00027, ips: 47.6485 samples/sec | ETA 01:18:24
2022-09-07 12:31:38 [INFO] [TRAIN] epoch: 49, iter: 4620/13950, loss: 0.1259, DSC: 88.3175, lr: 0.006963, batch_cost: 0.5023, reader_cost: 0.00026, ips: 47.7800 samples/sec | ETA 01:18:06
2022-09-07 12:31:44 [INFO] [TRAIN] epoch: 49, iter: 4630/13950, loss: 0.1331, DSC: 87.5747, lr: 0.006957, batch_cost: 0.5024, reader_cost: 0.00026, ips: 47.7719 samples/sec | ETA 01:18:02
2022-09-07 12:31:49 [INFO] [TRAIN] epoch: 49, iter: 4640/13950, loss: 0.1080, DSC: 90.0096, lr: 0.006950, batch_cost: 0.5046, reader_cost: 0.00027, ips: 47.5588 samples/sec | ETA 01:18:18
2022-09-07 12:31:53 [INFO] [TRAIN] epoch: 50, iter: 4650/13950, loss: 0.1900, DSC: 81.6874, lr: 0.006943, batch_cost: 0.4741, reader_cost: 0.00031, ips: 50.6226 samples/sec | ETA 01:13:29
2022-09-07 12:32:00 [INFO] [TRAIN] epoch: 50, iter: 4660/13950, loss: 0.1304, DSC: 87.8143, lr: 0.006936, batch_cost: 0.6410, reader_cost: 0.12859, ips: 37.4433 samples/sec | ETA 01:39:14
2022-09-07 12:32:05 [INFO] [TRAIN] epoch: 50, iter: 4670/13950, loss: 0.1122, DSC: 89.6044, lr: 0.006930, batch_cost: 0.5011, reader_cost: 0.00027, ips: 47.8939 samples/sec | ETA 01:17:30
2022-09-07 12:32:10 [INFO] [TRAIN] epoch: 50, iter: 4680/13950, loss: 0.1135, DSC: 89.4361, lr: 0.006923, batch_cost: 0.5030, reader_cost: 0.00028, ips: 47.7125 samples/sec | ETA 01:17:42
2022-09-07 12:32:15 [INFO] [TRAIN] epoch: 50, iter: 4690/13950, loss: 0.1174, DSC: 89.1839, lr: 0.006916, batch_cost: 0.5051, reader_cost: 0.00029, ips: 47.5152 samples/sec | ETA 01:17:57
2022-09-07 12:32:20 [INFO] [TRAIN] epoch: 50, iter: 4700/13950, loss: 0.1394, DSC: 86.9053, lr: 0.006910, batch_cost: 0.5036, reader_cost: 0.00027, ips: 47.6606 samples/sec | ETA 01:17:37
2022-09-07 12:32:25 [INFO] [TRAIN] epoch: 50, iter: 4710/13950, loss: 0.1168, DSC: 89.0578, lr: 0.006903, batch_cost: 0.5044, reader_cost: 0.00027, ips: 47.5802 samples/sec | ETA 01:17:40
2022-09-07 12:32:30 [INFO] [TRAIN] epoch: 50, iter: 4720/13950, loss: 0.1301, DSC: 87.8764, lr: 0.006896, batch_cost: 0.5038, reader_cost: 0.00026, ips: 47.6367 samples/sec | ETA 01:17:30
2022-09-07 12:32:35 [INFO] [TRAIN] epoch: 50, iter: 4730/13950, loss: 0.1287, DSC: 88.0521, lr: 0.006889, batch_cost: 0.5034, reader_cost: 0.00026, ips: 47.6797 samples/sec | ETA 01:17:20
2022-09-07 12:32:40 [INFO] [TRAIN] epoch: 50, iter: 4740/13950, loss: 0.1109, DSC: 89.8262, lr: 0.006883, batch_cost: 0.5017, reader_cost: 0.00023, ips: 47.8327 samples/sec | ETA 01:17:01
2022-09-07 12:32:46 [INFO] [TRAIN] epoch: 51, iter: 4750/13950, loss: 0.1816, DSC: 82.5200, lr: 0.006876, batch_cost: 0.6009, reader_cost: 0.11520, ips: 39.9383 samples/sec | ETA 01:32:08
2022-09-07 12:32:51 [INFO] [TRAIN] epoch: 51, iter: 4760/13950, loss: 0.1155, DSC: 89.3543, lr: 0.006869, batch_cost: 0.5078, reader_cost: 0.00029, ips: 47.2668 samples/sec | ETA 01:17:46
2022-09-07 12:32:56 [INFO] [TRAIN] epoch: 51, iter: 4770/13950, loss: 0.1195, DSC: 88.9154, lr: 0.006863, batch_cost: 0.5071, reader_cost: 0.00029, ips: 47.3274 samples/sec | ETA 01:17:35
2022-09-07 12:33:01 [INFO] [TRAIN] epoch: 51, iter: 4780/13950, loss: 0.1189, DSC: 88.9809, lr: 0.006856, batch_cost: 0.5058, reader_cost: 0.00028, ips: 47.4481 samples/sec | ETA 01:17:18
2022-09-07 12:33:06 [INFO] [TRAIN] epoch: 51, iter: 4790/13950, loss: 0.1432, DSC: 86.4785, lr: 0.006849, batch_cost: 0.5055, reader_cost: 0.00035, ips: 47.4812 samples/sec | ETA 01:17:10
2022-09-07 12:33:11 [INFO] [TRAIN] epoch: 51, iter: 4800/13950, loss: 0.1303, DSC: 87.7911, lr: 0.006842, batch_cost: 0.5027, reader_cost: 0.00025, ips: 47.7390 samples/sec | ETA 01:16:40
2022-09-07 12:33:16 [INFO] [TRAIN] epoch: 51, iter: 4810/13950, loss: 0.1259, DSC: 88.2198, lr: 0.006836, batch_cost: 0.5053, reader_cost: 0.00032, ips: 47.5008 samples/sec | ETA 01:16:58
2022-09-07 12:33:21 [INFO] [TRAIN] epoch: 51, iter: 4820/13950, loss: 0.1208, DSC: 89.0904, lr: 0.006829, batch_cost: 0.5043, reader_cost: 0.00025, ips: 47.5924 samples/sec | ETA 01:16:44
2022-09-07 12:33:26 [INFO] [TRAIN] epoch: 51, iter: 4830/13950, loss: 0.1339, DSC: 87.6279, lr: 0.006822, batch_cost: 0.5051, reader_cost: 0.00026, ips: 47.5152 samples/sec | ETA 01:16:46
2022-09-07 12:33:33 [INFO] [TRAIN] epoch: 52, iter: 4840/13950, loss: 0.1233, DSC: 88.6135, lr: 0.006815, batch_cost: 0.6222, reader_cost: 0.13101, ips: 38.5741 samples/sec | ETA 01:34:28
2022-09-07 12:33:38 [INFO] [TRAIN] epoch: 52, iter: 4850/13950, loss: 0.1326, DSC: 87.6379, lr: 0.006809, batch_cost: 0.5049, reader_cost: 0.00033, ips: 47.5349 samples/sec | ETA 01:16:34
2022-09-07 12:33:43 [INFO] [TRAIN] epoch: 52, iter: 4860/13950, loss: 0.1232, DSC: 88.6267, lr: 0.006802, batch_cost: 0.5037, reader_cost: 0.00026, ips: 47.6459 samples/sec | ETA 01:16:18
2022-09-07 12:33:48 [INFO] [TRAIN] epoch: 52, iter: 4870/13950, loss: 0.0959, DSC: 91.2821, lr: 0.006795, batch_cost: 0.5073, reader_cost: 0.00029, ips: 47.3054 samples/sec | ETA 01:16:46
2022-09-07 12:33:53 [INFO] [TRAIN] epoch: 52, iter: 4880/13950, loss: 0.1328, DSC: 87.5282, lr: 0.006788, batch_cost: 0.5086, reader_cost: 0.00034, ips: 47.1859 samples/sec | ETA 01:16:53
2022-09-07 12:33:58 [INFO] [TRAIN] epoch: 52, iter: 4890/13950, loss: 0.1320, DSC: 87.5808, lr: 0.006782, batch_cost: 0.5031, reader_cost: 0.00027, ips: 47.7025 samples/sec | ETA 01:15:58
2022-09-07 12:34:03 [INFO] [TRAIN] epoch: 52, iter: 4900/13950, loss: 0.1120, DSC: 89.6906, lr: 0.006775, batch_cost: 0.5038, reader_cost: 0.00027, ips: 47.6392 samples/sec | ETA 01:15:59
2022-09-07 12:34:08 [INFO] [TRAIN] epoch: 52, iter: 4910/13950, loss: 0.1122, DSC: 89.7176, lr: 0.006768, batch_cost: 0.5054, reader_cost: 0.00035, ips: 47.4912 samples/sec | ETA 01:16:08
2022-09-07 12:34:13 [INFO] [TRAIN] epoch: 52, iter: 4920/13950, loss: 0.1252, DSC: 88.3525, lr: 0.006762, batch_cost: 0.5043, reader_cost: 0.00029, ips: 47.5905 samples/sec | ETA 01:15:53
2022-09-07 12:34:19 [INFO] [TRAIN] epoch: 53, iter: 4930/13950, loss: 0.2004, DSC: 80.6856, lr: 0.006755, batch_cost: 0.5902, reader_cost: 0.11231, ips: 40.6666 samples/sec | ETA 01:28:43
2022-09-07 12:34:24 [INFO] [TRAIN] epoch: 53, iter: 4940/13950, loss: 0.1213, DSC: 88.7102, lr: 0.006748, batch_cost: 0.5207, reader_cost: 0.00035, ips: 46.0935 samples/sec | ETA 01:18:11
2022-09-07 12:34:29 [INFO] [TRAIN] epoch: 53, iter: 4950/13950, loss: 0.1081, DSC: 89.9496, lr: 0.006741, batch_cost: 0.5018, reader_cost: 0.00026, ips: 47.8322 samples/sec | ETA 01:15:15
2022-09-07 12:34:34 [INFO] [TRAIN] epoch: 53, iter: 4960/13950, loss: 0.1312, DSC: 87.8612, lr: 0.006735, batch_cost: 0.5038, reader_cost: 0.00027, ips: 47.6423 samples/sec | ETA 01:15:28
2022-09-07 12:34:39 [INFO] [TRAIN] epoch: 53, iter: 4970/13950, loss: 0.1160, DSC: 89.1131, lr: 0.006728, batch_cost: 0.5026, reader_cost: 0.00025, ips: 47.7491 samples/sec | ETA 01:15:13
2022-09-07 12:34:44 [INFO] [TRAIN] epoch: 53, iter: 4980/13950, loss: 0.1146, DSC: 89.4022, lr: 0.006721, batch_cost: 0.5033, reader_cost: 0.00026, ips: 47.6867 samples/sec | ETA 01:15:14
2022-09-07 12:34:49 [INFO] [TRAIN] epoch: 53, iter: 4990/13950, loss: 0.1219, DSC: 88.6015, lr: 0.006714, batch_cost: 0.5023, reader_cost: 0.00026, ips: 47.7830 samples/sec | ETA 01:15:00
2022-09-07 12:34:54 [INFO] [TRAIN] epoch: 53, iter: 5000/13950, loss: 0.1193, DSC: 88.9187, lr: 0.006708, batch_cost: 0.5129, reader_cost: 0.00029, ips: 46.7904 samples/sec | ETA 01:16:30
2022-09-07 12:34:54 [INFO] Start evaluating (total_samples: 12, total_iters: 12)...
1/12 [=>............................] - ETA: 22s - batch_cost: 2.0131 - reader cost: 1.1657 2/12 [====>.........................] - ETA: 12s - batch_cost: 1.2735 - reader cost: 0.5856 3/12 [======>.......................] - ETA: 9s - batch_cost: 1.0494 - reader cost: 0.3981 4/12 [=========>....................] - ETA: 9s - batch_cost: 1.2057 - reader cost: 0.4552 5/12 [===========>..................] - ETA: 7s - batch_cost: 1.1361 - reader cost: 0.3684 6/12 [==============>...............] - ETA: 6s - batch_cost: 1.0849 - reader cost: 0.3135 7/12 [================>.............] - ETA: 5s - batch_cost: 1.0158 - reader cost: 0.2714 8/12 [===================>..........] - ETA: 4s - batch_cost: 1.1157 - reader cost: 0.3197 9/12 [=====================>........] - ETA: 3s - batch_cost: 1.0894 - reader cost: 0.286410/12 [========================>.....] - ETA: 2s - batch_cost: 1.0639 - reader cost: 0.260811/12 [==========================>...] - ETA: 1s - batch_cost: 1.0132 - reader cost: 0.238612/12 [==============================] - 12s 976ms/step - batch_cost: 0.9758 - reader cost: 0.2204
2022-09-07 12:35:06 [INFO] [EVAL] #Images: 12, Dice: 0.8185, Loss: 0.200956
2022-09-07 12:35:06 [INFO] [EVAL] Class dice:
[0.995 0.8946 0.527 0.881 0.8472 0.946 0.6372 0.8768 0.7618]
2022-09-07 12:35:12 [INFO] [EVAL] The model with the best validation mDice (0.8185) was saved at iter 5000.
2022-09-07 12:35:17 [INFO] [TRAIN] epoch: 53, iter: 5010/13950, loss: 0.1111, DSC: 89.7339, lr: 0.006701, batch_cost: 0.5013, reader_cost: 0.00037, ips: 47.8743 samples/sec | ETA 01:14:41
2022-09-07 12:35:22 [INFO] [TRAIN] epoch: 53, iter: 5020/13950, loss: 0.1329, DSC: 87.6178, lr: 0.006694, batch_cost: 0.4980, reader_cost: 0.00021, ips: 48.1929 samples/sec | ETA 01:14:07
2022-09-07 12:35:28 [INFO] [TRAIN] epoch: 54, iter: 5030/13950, loss: 0.1732, DSC: 83.5166, lr: 0.006687, batch_cost: 0.6419, reader_cost: 0.15545, ips: 37.3914 samples/sec | ETA 01:35:25
2022-09-07 12:35:33 [INFO] [TRAIN] epoch: 54, iter: 5040/13950, loss: 0.1171, DSC: 89.1591, lr: 0.006681, batch_cost: 0.5033, reader_cost: 0.00033, ips: 47.6830 samples/sec | ETA 01:14:44
2022-09-07 12:35:38 [INFO] [TRAIN] epoch: 54, iter: 5050/13950, loss: 0.1153, DSC: 89.1735, lr: 0.006674, batch_cost: 0.5027, reader_cost: 0.00026, ips: 47.7386 samples/sec | ETA 01:14:34
2022-09-07 12:35:43 [INFO] [TRAIN] epoch: 54, iter: 5060/13950, loss: 0.1255, DSC: 88.1794, lr: 0.006667, batch_cost: 0.5033, reader_cost: 0.00025, ips: 47.6892 samples/sec | ETA 01:14:33
2022-09-07 12:35:49 [INFO] [TRAIN] epoch: 54, iter: 5070/13950, loss: 0.1241, DSC: 88.5502, lr: 0.006660, batch_cost: 0.5041, reader_cost: 0.00033, ips: 47.6057 samples/sec | ETA 01:14:36
2022-09-07 12:35:54 [INFO] [TRAIN] epoch: 54, iter: 5080/13950, loss: 0.1265, DSC: 88.2408, lr: 0.006654, batch_cost: 0.5033, reader_cost: 0.00028, ips: 47.6865 samples/sec | ETA 01:14:24
2022-09-07 12:35:59 [INFO] [TRAIN] epoch: 54, iter: 5090/13950, loss: 0.1244, DSC: 88.4840, lr: 0.006647, batch_cost: 0.5363, reader_cost: 0.00029, ips: 44.7487 samples/sec | ETA 01:19:11
2022-09-07 12:36:04 [INFO] [TRAIN] epoch: 54, iter: 5100/13950, loss: 0.1368, DSC: 87.2265, lr: 0.006640, batch_cost: 0.5054, reader_cost: 0.00027, ips: 47.4915 samples/sec | ETA 01:14:32
2022-09-07 12:36:09 [INFO] [TRAIN] epoch: 54, iter: 5110/13950, loss: 0.1348, DSC: 87.3528, lr: 0.006633, batch_cost: 0.5035, reader_cost: 0.00025, ips: 47.6662 samples/sec | ETA 01:14:10
2022-09-07 12:36:15 [INFO] [TRAIN] epoch: 55, iter: 5120/13950, loss: 0.1364, DSC: 87.2225, lr: 0.006627, batch_cost: 0.5835, reader_cost: 0.09667, ips: 41.1288 samples/sec | ETA 01:25:52
2022-09-07 12:36:20 [INFO] [TRAIN] epoch: 55, iter: 5130/13950, loss: 0.1190, DSC: 88.9011, lr: 0.006620, batch_cost: 0.5036, reader_cost: 0.00026, ips: 47.6572 samples/sec | ETA 01:14:01
2022-09-07 12:36:25 [INFO] [TRAIN] epoch: 55, iter: 5140/13950, loss: 0.1356, DSC: 87.4000, lr: 0.006613, batch_cost: 0.5052, reader_cost: 0.00034, ips: 47.5015 samples/sec | ETA 01:14:11
2022-09-07 12:36:30 [INFO] [TRAIN] epoch: 55, iter: 5150/13950, loss: 0.1094, DSC: 89.9522, lr: 0.006606, batch_cost: 0.5041, reader_cost: 0.00028, ips: 47.6082 samples/sec | ETA 01:13:56
2022-09-07 12:36:35 [INFO] [TRAIN] epoch: 55, iter: 5160/13950, loss: 0.1246, DSC: 88.2818, lr: 0.006600, batch_cost: 0.5056, reader_cost: 0.00035, ips: 47.4721 samples/sec | ETA 01:14:03
2022-09-07 12:36:40 [INFO] [TRAIN] epoch: 55, iter: 5170/13950, loss: 0.1364, DSC: 87.2114, lr: 0.006593, batch_cost: 0.5054, reader_cost: 0.00027, ips: 47.4849 samples/sec | ETA 01:13:57
2022-09-07 12:36:45 [INFO] [TRAIN] epoch: 55, iter: 5180/13950, loss: 0.1153, DSC: 89.3343, lr: 0.006586, batch_cost: 0.5051, reader_cost: 0.00027, ips: 47.5187 samples/sec | ETA 01:13:49
2022-09-07 12:36:50 [INFO] [TRAIN] epoch: 55, iter: 5190/13950, loss: 0.1100, DSC: 89.7588, lr: 0.006579, batch_cost: 0.5050, reader_cost: 0.00026, ips: 47.5222 samples/sec | ETA 01:13:44
2022-09-07 12:36:55 [INFO] [TRAIN] epoch: 55, iter: 5200/13950, loss: 0.1144, DSC: 89.4644, lr: 0.006573, batch_cost: 0.5048, reader_cost: 0.00034, ips: 47.5396 samples/sec | ETA 01:13:37
2022-09-07 12:37:01 [INFO] [TRAIN] epoch: 56, iter: 5210/13950, loss: 0.1526, DSC: 85.6447, lr: 0.006566, batch_cost: 0.6026, reader_cost: 0.11727, ips: 39.8265 samples/sec | ETA 01:27:46
2022-09-07 12:37:06 [INFO] [TRAIN] epoch: 56, iter: 5220/13950, loss: 0.1172, DSC: 89.1067, lr: 0.006559, batch_cost: 0.5114, reader_cost: 0.00027, ips: 46.9325 samples/sec | ETA 01:14:24
2022-09-07 12:37:11 [INFO] [TRAIN] epoch: 56, iter: 5230/13950, loss: 0.1186, DSC: 89.0139, lr: 0.006552, batch_cost: 0.5044, reader_cost: 0.00035, ips: 47.5788 samples/sec | ETA 01:13:18
2022-09-07 12:37:17 [INFO] [TRAIN] epoch: 56, iter: 5240/13950, loss: 0.1183, DSC: 88.9207, lr: 0.006546, batch_cost: 0.5022, reader_cost: 0.00026, ips: 47.7894 samples/sec | ETA 01:12:54
2022-09-07 12:37:22 [INFO] [TRAIN] epoch: 56, iter: 5250/13950, loss: 0.1231, DSC: 88.5389, lr: 0.006539, batch_cost: 0.5043, reader_cost: 0.00041, ips: 47.5873 samples/sec | ETA 01:13:07
2022-09-07 12:37:27 [INFO] [TRAIN] epoch: 56, iter: 5260/13950, loss: 0.1155, DSC: 89.3077, lr: 0.006532, batch_cost: 0.5028, reader_cost: 0.00025, ips: 47.7348 samples/sec | ETA 01:12:49
2022-09-07 12:37:32 [INFO] [TRAIN] epoch: 56, iter: 5270/13950, loss: 0.1135, DSC: 89.4213, lr: 0.006525, batch_cost: 0.5032, reader_cost: 0.00027, ips: 47.6968 samples/sec | ETA 01:12:47
2022-09-07 12:37:37 [INFO] [TRAIN] epoch: 56, iter: 5280/13950, loss: 0.1111, DSC: 89.7731, lr: 0.006518, batch_cost: 0.5036, reader_cost: 0.00026, ips: 47.6572 samples/sec | ETA 01:12:46
2022-09-07 12:37:42 [INFO] [TRAIN] epoch: 56, iter: 5290/13950, loss: 0.1323, DSC: 87.6354, lr: 0.006512, batch_cost: 0.5050, reader_cost: 0.00033, ips: 47.5264 samples/sec | ETA 01:12:53
2022-09-07 12:37:47 [INFO] [TRAIN] epoch: 56, iter: 5300/13950, loss: 0.1294, DSC: 87.8431, lr: 0.006505, batch_cost: 0.5032, reader_cost: 0.00022, ips: 47.6901 samples/sec | ETA 01:12:33
2022-09-07 12:37:53 [INFO] [TRAIN] epoch: 57, iter: 5310/13950, loss: 0.1902, DSC: 81.8065, lr: 0.006498, batch_cost: 0.6062, reader_cost: 0.11881, ips: 39.5918 samples/sec | ETA 01:27:17
2022-09-07 12:37:58 [INFO] [TRAIN] epoch: 57, iter: 5320/13950, loss: 0.1322, DSC: 87.7281, lr: 0.006491, batch_cost: 0.5064, reader_cost: 0.00027, ips: 47.3901 samples/sec | ETA 01:12:50
2022-09-07 12:38:03 [INFO] [TRAIN] epoch: 57, iter: 5330/13950, loss: 0.1326, DSC: 87.7314, lr: 0.006485, batch_cost: 0.5059, reader_cost: 0.00033, ips: 47.4440 samples/sec | ETA 01:12:40
2022-09-07 12:38:08 [INFO] [TRAIN] epoch: 57, iter: 5340/13950, loss: 0.1281, DSC: 88.0884, lr: 0.006478, batch_cost: 0.5038, reader_cost: 0.00027, ips: 47.6378 samples/sec | ETA 01:12:17
2022-09-07 12:38:13 [INFO] [TRAIN] epoch: 57, iter: 5350/13950, loss: 0.1234, DSC: 88.5145, lr: 0.006471, batch_cost: 0.5047, reader_cost: 0.00027, ips: 47.5497 samples/sec | ETA 01:12:20
2022-09-07 12:38:18 [INFO] [TRAIN] epoch: 57, iter: 5360/13950, loss: 0.1397, DSC: 86.9090, lr: 0.006464, batch_cost: 0.5050, reader_cost: 0.00027, ips: 47.5207 samples/sec | ETA 01:12:18
2022-09-07 12:38:23 [INFO] [TRAIN] epoch: 57, iter: 5370/13950, loss: 0.1188, DSC: 89.0113, lr: 0.006458, batch_cost: 0.5039, reader_cost: 0.00027, ips: 47.6265 samples/sec | ETA 01:12:03
2022-09-07 12:38:28 [INFO] [TRAIN] epoch: 57, iter: 5380/13950, loss: 0.1260, DSC: 88.2354, lr: 0.006451, batch_cost: 0.5040, reader_cost: 0.00033, ips: 47.6200 samples/sec | ETA 01:11:59
2022-09-07 12:38:33 [INFO] [TRAIN] epoch: 57, iter: 5390/13950, loss: 0.1289, DSC: 87.8548, lr: 0.006444, batch_cost: 0.5037, reader_cost: 0.00025, ips: 47.6507 samples/sec | ETA 01:11:51
2022-09-07 12:38:39 [INFO] [TRAIN] epoch: 58, iter: 5400/13950, loss: 0.1832, DSC: 82.4263, lr: 0.006437, batch_cost: 0.5991, reader_cost: 0.11432, ips: 40.0582 samples/sec | ETA 01:25:22
2022-09-07 12:38:44 [INFO] [TRAIN] epoch: 58, iter: 5410/13950, loss: 0.1038, DSC: 90.5556, lr: 0.006430, batch_cost: 0.5019, reader_cost: 0.00026, ips: 47.8173 samples/sec | ETA 01:11:26
2022-09-07 12:38:49 [INFO] [TRAIN] epoch: 58, iter: 5420/13950, loss: 0.1444, DSC: 86.3499, lr: 0.006424, batch_cost: 0.5009, reader_cost: 0.00027, ips: 47.9110 samples/sec | ETA 01:11:12
2022-09-07 12:38:54 [INFO] [TRAIN] epoch: 58, iter: 5430/13950, loss: 0.1086, DSC: 89.9062, lr: 0.006417, batch_cost: 0.5047, reader_cost: 0.00029, ips: 47.5516 samples/sec | ETA 01:11:40
2022-09-07 12:38:59 [INFO] [TRAIN] epoch: 58, iter: 5440/13950, loss: 0.1153, DSC: 89.3315, lr: 0.006410, batch_cost: 0.5039, reader_cost: 0.00026, ips: 47.6238 samples/sec | ETA 01:11:28
2022-09-07 12:39:04 [INFO] [TRAIN] epoch: 58, iter: 5450/13950, loss: 0.1191, DSC: 89.0811, lr: 0.006403, batch_cost: 0.5046, reader_cost: 0.00027, ips: 47.5662 samples/sec | ETA 01:11:28
2022-09-07 12:39:09 [INFO] [TRAIN] epoch: 58, iter: 5460/13950, loss: 0.1397, DSC: 86.7790, lr: 0.006397, batch_cost: 0.5044, reader_cost: 0.00032, ips: 47.5782 samples/sec | ETA 01:11:22
2022-09-07 12:39:14 [INFO] [TRAIN] epoch: 58, iter: 5470/13950, loss: 0.1363, DSC: 87.1961, lr: 0.006390, batch_cost: 0.5044, reader_cost: 0.00028, ips: 47.5771 samples/sec | ETA 01:11:17
2022-09-07 12:39:19 [INFO] [TRAIN] epoch: 58, iter: 5480/13950, loss: 0.1097, DSC: 89.8432, lr: 0.006383, batch_cost: 0.5032, reader_cost: 0.00025, ips: 47.6980 samples/sec | ETA 01:11:01
2022-09-07 12:39:25 [INFO] [TRAIN] epoch: 59, iter: 5490/13950, loss: 0.1800, DSC: 82.7680, lr: 0.006376, batch_cost: 0.5897, reader_cost: 0.10422, ips: 40.6980 samples/sec | ETA 01:23:08
2022-09-07 12:39:30 [INFO] [TRAIN] epoch: 59, iter: 5500/13950, loss: 0.1162, DSC: 89.2048, lr: 0.006369, batch_cost: 0.5047, reader_cost: 0.00026, ips: 47.5527 samples/sec | ETA 01:11:04
2022-09-07 12:39:35 [INFO] [TRAIN] epoch: 59, iter: 5510/13950, loss: 0.1153, DSC: 89.2791, lr: 0.006363, batch_cost: 0.5032, reader_cost: 0.00027, ips: 47.6924 samples/sec | ETA 01:10:47
2022-09-07 12:39:40 [INFO] [TRAIN] epoch: 59, iter: 5520/13950, loss: 0.1131, DSC: 89.5481, lr: 0.006356, batch_cost: 0.5042, reader_cost: 0.00026, ips: 47.6007 samples/sec | ETA 01:10:50