Reload weights after plateau

[DanBmh]

Reload checkpoint weights after reaching a plateau that we use the best_dev weights again

[community-tc-integration]

No Taskcluster jobs started for this pull request

The `allowPullRequests` configuration for this repository (in `.taskcluster.yml` on the
default branch) does not allow starting tasks for this pull request.

[lissyx]

@DanBmh Thanks, can you elaborate a little bit? My mind is kind of somewhere else, so I'm unsure I get the point here.

[DanBmh]

Currently training looks like this:

epoch 5: val_loss=62
e6: vl=59
e7: vl=60
e8: vl=61
Reached a plateau, LearningRate:=LR*0.1
e9: vl=60       <- Here we're using the weights from e8, with the suggested changes we're using e6 instead
e10: vl=58   <- We have an improvement but the network has to do some more work to fix the errors from e7+e8

The old approach did still work well but I think we can make it even better by reloading the weights from the best_dev checkpoint.

[DanBmh]

Not ready yet!
Found an error when using --drop_source_layers flag

[DanBmh]

Working again:)

[lissyx]

LGTM, I'd like Reuben's opinion.

[reuben]

Please keep this function unchanged and add a new explicitly load_best_checkpoint function, we don't want this call to load last silently for example.

[reuben]

Re-opened as #3261 to run tests.

[reuben]

Merged in #3261