Train on 2561 samples, val on 284 samples, with batch size 32. Epoch 1/50 2019-09-13 08:40:49.176893: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library cublas64_100.dll locally 80/80 [==============================] - 131s 2s/step - loss: 517.3584 - val_loss: 82.3485 Epoch 2/50 80/80 [==============================] - 130s 2s/step - loss: 55.8335 - val_loss: 44.6843 Epoch 3/50 80/80 [==============================] - 131s 2s/step - loss: 37.6996 - val_loss: 35.8303 Epoch 4/50 80/80 [==============================] - 130s 2s/step - loss: 31.1519 - val_loss: 28.8108 Epoch 5/50 80/80 [==============================] - 131s 2s/step - loss: 27.9181 - val_loss: 27.9306 Epoch 6/50 80/80 [==============================] - 128s 2s/step - loss: 25.9196 - val_loss: 25.1906 Epoch 7/50 80/80 [==============================] - 128s 2s/step - loss: 24.6425 - val_loss: 24.7121 Epoch 8/50 80/80 [==============================] - 128s 2s/step - loss: 23.5448 - val_loss: 24.5065 Epoch 9/50 80/80 [==============================] - 128s 2s/step - loss: 22.9092 - val_loss: 22.7452 Epoch 10/50 80/80 [==============================] - 128s 2s/step - loss: 22.0492 - val_loss: 22.7810 Epoch 11/50 80/80 [==============================] - 128s 2s/step - loss: 21.4386 - val_loss: 21.8725 Epoch 12/50 80/80 [==============================] - 128s 2s/step - loss: 21.0675 - val_loss: 20.6727 Epoch 13/50 80/80 [==============================] - 128s 2s/step - loss: 20.9114 - val_loss: 21.4525 Epoch 14/50 80/80 [==============================] - 127s 2s/step - loss: 20.5407 - val_loss: 21.0128 Epoch 15/50 80/80 [==============================] - 127s 2s/step - loss: 20.2319 - val_loss: 19.0574 Epoch 16/50 80/80 [==============================] - 128s 2s/step - loss: 19.7329 - val_loss: 21.7510 Epoch 17/50 80/80 [==============================] - 129s 2s/step - loss: 19.3586 - val_loss: 19.4242 Epoch 18/50 80/80 [==============================] - 129s 2s/step - loss: 19.3137 - val_loss: 19.2081 Epoch 19/50 80/80 [==============================] - 129s 2s/step - loss: 19.1857 - val_loss: 19.2701 Epoch 20/50 80/80 [==============================] - 129s 2s/step - loss: 18.8003 - val_loss: 19.7174 Epoch 21/50 80/80 [==============================] - 130s 2s/step - loss: 18.6989 - val_loss: 18.8449 Epoch 22/50 80/80 [==============================] - 130s 2s/step - loss: 18.4523 - val_loss: 19.1494 Epoch 23/50 80/80 [==============================] - 130s 2s/step - loss: 18.1344 - val_loss: 18.1966 Epoch 24/50 80/80 [==============================] - 130s 2s/step - loss: 17.8841 - val_loss: 17.9396 Epoch 25/50 80/80 [==============================] - 130s 2s/step - loss: 17.7208 - val_loss: 17.9037 Epoch 26/50 80/80 [==============================] - 130s 2s/step - loss: 17.5462 - val_loss: 18.0783 Epoch 27/50 80/80 [==============================] - 130s 2s/step - loss: 17.5054 - val_loss: 18.4249 Epoch 28/50 80/80 [==============================] - 131s 2s/step - loss: 17.2148 - val_loss: 17.3287 Epoch 29/50 80/80 [==============================] - 131s 2s/step - loss: 17.3045 - val_loss: 17.8810 Epoch 30/50 80/80 [==============================] - 131s 2s/step - loss: 16.7447 - val_loss: 17.2171 Epoch 31/50 80/80 [==============================] - 132s 2s/step - loss: 16.6400 - val_loss: 17.4705 Epoch 32/50 80/80 [==============================] - 135s 2s/step - loss: 16.6576 - val_loss: 16.5528 Epoch 33/50 80/80 [==============================] - 144s 2s/step - loss: 16.7870 - val_loss: 16.9810 Epoch 34/50 80/80 [==============================] - 139s 2s/step - loss: 16.3223 - val_loss: 17.0919 Epoch 35/50 80/80 [==============================] - 133s 2s/step - loss: 16.4223 - val_loss: 16.3638 Epoch 36/50 80/80 [==============================] - 132s 2s/step - loss: 16.0801 - val_loss: 16.3114 Epoch 37/50 80/80 [==============================] - 133s 2s/step - loss: 16.2183 - val_loss: 15.9354 Epoch 38/50 80/80 [==============================] - 133s 2s/step - loss: 15.9897 - val_loss: 16.0493 Epoch 39/50 80/80 [==============================] - 133s 2s/step - loss: 16.0378 - val_loss: 17.0315 Epoch 40/50 80/80 [==============================] - 134s 2s/step - loss: 15.5596 - val_loss: 16.0732 Epoch 41/50 80/80 [==============================] - 146s 2s/step - loss: 15.6794 - val_loss: 15.9830 Epoch 42/50 80/80 [==============================] - 139s 2s/step - loss: 15.7063 - val_loss: 15.5648 Epoch 43/50 80/80 [==============================] - 137s 2s/step - loss: 15.5948 - val_loss: 16.4656 Epoch 44/50 80/80 [==============================] - 134s 2s/step - loss: 15.4423 - val_loss: 17.1915 Epoch 45/50 80/80 [==============================] - 134s 2s/step - loss: 15.3306 - val_loss: 14.3016 Epoch 46/50 80/80 [==============================] - 134s 2s/step - loss: 15.4073 - val_loss: 15.9408 Epoch 47/50 80/80 [==============================] - 134s 2s/step - loss: 15.3174 - val_loss: 16.1294 Epoch 48/50 80/80 [==============================] - 131s 2s/step - loss: 15.1787 - val_loss: 15.3598 Epoch 49/50 80/80 [==============================] - 128s 2s/step - loss: 15.1787 - val_loss: 15.1178 Epoch 50/50 80/80 [==============================] - 128s 2s/step - loss: 15.1992 - val_loss: 15.0141 Unfreeze all of the layers. Train on 2561 samples, val on 284 samples, with batch size 32. Epoch 51/100 80/80 [==============================] - 131s 2s/step - loss: 12.8788 - val_loss: 12.0782 Epoch 52/100 80/80 [==============================] - 129s 2s/step - loss: 12.2756 - val_loss: 11.8784 Epoch 53/100 80/80 [==============================] - 129s 2s/step - loss: 11.8105 - val_loss: 11.2780 Epoch 54/100 80/80 [==============================] - 129s 2s/step - loss: 11.4993 - val_loss: 11.2301 Epoch 55/100 80/80 [==============================] - 130s 2s/step - loss: 11.3352 - val_loss: 10.8529 Epoch 56/100 80/80 [==============================] - 129s 2s/step - loss: 11.0819 - val_loss: 10.9545 Epoch 57/100 80/80 [==============================] - 129s 2s/step - loss: 11.0293 - val_loss: 10.9453 Epoch 58/100 80/80 [==============================] - 129s 2s/step - loss: 10.8480 - val_loss: 11.0160
Epoch 00058: ReduceLROnPlateau reducing learning rate to 9.999999747378752e-06. Epoch 59/100 80/80 [==============================] - 129s 2s/step - loss: 10.7110 - val_loss: 10.6123 Epoch 60/100 80/80 [==============================] - 129s 2s/step - loss: 10.5758 - val_loss: 10.7624 Epoch 61/100 80/80 [==============================] - 130s 2s/step - loss: 10.7129 - val_loss: 10.5631 Epoch 62/100 80/80 [==============================] - 130s 2s/step - loss: 10.6693 - val_loss: 10.4871 Epoch 63/100 80/80 [==============================] - 130s 2s/step - loss: 10.5558 - val_loss: 10.4733 Epoch 64/100 80/80 [==============================] - 130s 2s/step - loss: 10.5404 - val_loss: 10.7139 Epoch 65/100 80/80 [==============================] - 130s 2s/step - loss: 10.5845 - val_loss: 10.2280 Epoch 66/100 80/80 [==============================] - 130s 2s/step - loss: 10.6317 - val_loss: 10.2237 Epoch 67/100 80/80 [==============================] - 130s 2s/step - loss: 10.5670 - val_loss: 10.7456 Epoch 68/100 80/80 [==============================] - 130s 2s/step - loss: 10.4998 - val_loss: 10.1302 Epoch 69/100 80/80 [==============================] - 129s 2s/step - loss: 10.4921 - val_loss: 10.3209 Epoch 70/100 80/80 [==============================] - 130s 2s/step - loss: 10.4794 - val_loss: 10.3448 Epoch 71/100 80/80 [==============================] - 129s 2s/step - loss: 10.4624 - val_loss: 10.8767
Epoch 00071: ReduceLROnPlateau reducing learning rate to 9.999999747378752e-07. Epoch 72/100 80/80 [==============================] - 129s 2s/step - loss: 10.5583 - val_loss: 10.5669 Epoch 73/100 80/80 [==============================] - 130s 2s/step - loss: 10.4452 - val_loss: 9.9291 Epoch 74/100 80/80 [==============================] - 129s 2s/step - loss: 10.6423 - val_loss: 10.9893 Epoch 75/100 80/80 [==============================] - 129s 2s/step - loss: 10.4593 - val_loss: 10.3776 Epoch 76/100 80/80 [==============================] - 129s 2s/step - loss: 10.3698 - val_loss: 10.0692
Epoch 00076: ReduceLROnPlateau reducing learning rate to 9.999999974752428e-08. Epoch 77/100 80/80 [==============================] - 129s 2s/step - loss: 10.5463 - val_loss: 10.9560 Epoch 78/100 80/80 [==============================] - 129s 2s/step - loss: 10.5106 - val_loss: 10.4077 Epoch 79/100 80/80 [==============================] - 129s 2s/step - loss: 10.5886 - val_loss: 10.3315
Epoch 00079: ReduceLROnPlateau reducing learning rate to 1.0000000116860975e-08. Epoch 80/100 80/80 [==============================] - 130s 2s/step - loss: 10.4572 - val_loss: 10.6871 Epoch 81/100 80/80 [==============================] - 129s 2s/step - loss: 10.4784 - val_loss: 10.1432 Epoch 82/100 80/80 [==============================] - 129s 2s/step - loss: 10.4856 - val_loss: 10.7359
Epoch 00082: ReduceLROnPlateau reducing learning rate to 9.999999939225292e-10. Epoch 83/100 80/80 [==============================] - 130s 2s/step - loss: 10.6101 - val_loss: 11.2048 Epoch 00083: early stopping
(base) C:\Users\kzk_k\OneDrive\keras\yolov3\keras-yolov3>