Why BERT is suggesting to fine tune and not feature extraction?

[manohar1989]

I searched TFHUB and found out that BERT has to be fine tuned. All params to be updated, but i did not understand why bert should be fine tuned.
The features, bidirectional meaning of words would remain same for english language atleast.
I tried with and without freezing the layers, the output when freezed seems better for my usecase.

[mrdbourke]

Hey there,

Do you mind sharing the link where you’re reading this so I can learn more too?

There are many different models of BERT on TensorFlow Hub.

Though it seems like you’ve tried it and it’s working?

[manohar1989]

https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4
Last paragraph of Overview

In general my doubt is that for images we have feature extraction but for NLP (Bert, gpt etc) say fine tune. Even though embeddings can be fetched, but why specifically NLP models are being fine tuned. Is it because the language per domain changes in some way? This I am unable to confirm.

[mrdbourke]

Oh I see where you mean.

Looks like it can be used for feature extraction (first few paragraphs).

But the fine-tuning of the language model (the .mlm file) is an advanced optional extension.

Have you tried it as a feature extractor?

[manohar1989]

yes, i tried both feature extraction as well as fine tuning. After a bunch of experiments, I find both of them not really different from each other in my case.
just adding one layer on top gives better result. I see in many places people are calling the addition of one top layer as fine tuning in case of bert.

from : https://www.analyticsvidhya.com/blog/2020/07/transfer-learning-for-nlp-fine-tuning-bert-for-text-classification/

I think this is where i got confused.

[mrdbourke]

I see where the confusion could come from.

I personally wouldn't call adding a single layer on top of a trained architecture fine-tuning (in most cases, you're really only training 1 layer out of ~200+).

I'd call that feature extraction.

Glad to hear you got things working after experimenting though!