Re-use all weights when re-training on same dataset

[gheinrich]

Current behaviour:
When a classification model is being re-trained the last InnerProduct layer is renamed to allow training on another dataset (with a possibly different number of classes).
This implies that the weights of that last layer are discarded.

Problem:
If the model is being re-trained on the same dataset then initially the model will perform badly as it needs to re-learn the weights of its last InnerProduct layer.

Solution:
Refrain from renaming the last InnerProduct layer if the model is being re-trained on the same dataset

Note:
This applies to Caffe only.

closes #533

[lukeyeager]

Seems weird to have previous_network (with an underscore) right next to useSameDataset (using camelCase).

[gheinrich]

oh yes, I'll change that

[lukeyeager]

Looks like it works to me. Thanks for the fix!

[gheinrich]

Updated with underscore naming convention

[mfernezir]

Hi!

@gheinrich @lukeyeager there are still a couple of issues with this solution:

1.) There's another use case where one would not want to lose weights from the last layer: continuing training on a larger dataset after pretraining and experimenting on a smaller sample from the same training distribution of images. To be able to do that for my current task, I manually redefined the use_same_dataset variable to True in two Python files changed in this commit.

2.) Not all networks have only one inner product classification layer. In my case, I'm using GoogLeNet which has 3 outputs. At the moment, what happens is that the final classification layer gets erased when loading another dataset but two earlier ones don't.

Perhaps it would be better to always preserve pretrained weights and leave it to the user to manually rename layers that he wants trained from scratch?

[lukeyeager]

Good point @mfernezir - we don't have a good general-purpose solution yet. It's a tricky problem because we'd like to satisfy a bunch of conditions:

• Automatically rename and resize output layer[s] to match the number of classes
• Handle multiple output layers
• Handle output layer types which are not InnerProduct layers
• Use a similar approach for both Torch and Caffe

Since you can't have variables in a .prototxt file, I've considered adding some keywords to Caffe .prototxt files which can be search/replaced before processing with Caffe, like this:

inner_product_param {
  num_output: NUM_CLASSES
}

But I'd want a solution that enables Caffe to use the original .prototxt file without alteration, too. Maybe something in a comment like this?

inner_product_param {
  #replace with NUM_CLASSES#
  num_output: 1000
}

[mfernezir]

I'm not sure what would be best. In any case, I think that between conflicting design objectives transparency and getting expected behavior should be more important than achieving automation and convenience. One simple option would be the following:

1. Don't bother with automatic accommodation to different number of classes. It's not hard to add num_output wherever it is needed, if one knows that's required. Comments in standard nets definitions and documenting the requirement would suffice.
2. Emphasize Caffe requirement to rename all layers which are to be retrained when loading weights from a previous model. There is already a disclaimer in place that one should know how finetuning is done in Caffe before using the feature; it is more troublesome to unexpectedly get some weights preserved and some erased.