Modifying the "initial_epoch" arg when doing fine tuning and transfer learning

[skyblasy]

Hey all,

I'm having fun with the course and wanted to do some experimentation in terms of altering the number of layers to make trainable for the fine tuning section of transfer learning.

I had a bit of an issue and error message though when I altered the number of trainable errors, and it was indicating that I needed to modify the "inital_epoch" argument in the fit set. What is the general rule when it comes to that. I saw that? How do I modify that argument to accommodate some arbitrary amount of layers I want to train?

Thanks!
Sky

[mrdbourke]

Hey there,

Massive effort experimenting with different layers for fine-tuning.

As for the initial_epoch, that's more to do with what number epoch the model should resume training from when saved to disk.

See the docs: https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit

initial_epoch - Integer. Epoch at which to start training (useful for resuming a previous training run).

For example, if you saved a model and wanted to continue training, you could start training on the last epoch of the training run where it was saved (e.g. history.epoch[-1]).

If you wanted to go back further, you could use history.epoch[-5] for example.

When fine-tuning you might want to experiment unfreezing layers using layer.trainable = True and then going back a certain number of epochs to change how much training is done.

[skyblasy]

Ok, so it actually isn't necessary, per se, when doing fine tuning, but only if you've saved a model, correct?

Like, I could go through a modelling workflow, and (potentially) omit saving the model and I wouldn't need that argument in the fit step?

Thanks, Daniel.

[mrdbourke]

Yes, you're right, you really only need to use the initial epoch if you're loading in a saved model and continuing training.

However, if you start training a model again, its "number of epochs" in the history object will get reset unless you specify what number it started on.

Knowing the "number of epochs" a model has been trained for may be useful later on.

[john-james-ai]

Fantastic course! The notebooks are a go-to reference.

I have a related question.

What are the benefits and drawbacks of starting each fine-tuning session from the previous last epoch, versus reloading the feature extraction weights, thawing the n top layers of the base model, and then training from the last feature extraction epoch with the new model each time?

I'm currently experimenting with both approaches and will update with my observations. Meanwhile, I'd be interested in your perspectives.