[HPO] Enabling ote_anomalib report score for hpopt

[SangdaeNam]

This PR contains adding validation_epoch_end function in ProgressCallback in ote_anomalib.
In hpo flow, score in every validation epoch end report to hpopt with order below

1. In ProgressCallback, update_progress_callback is HPOCallback delivered from train_task in ote_anomalib
2. In every validation epoch end, the score produced by metric deliver to update_progress_callback
3. When update_progress_callback(which is HPOCallback) is called, it reports the score to hpopt
4. With the score delivered, hpopt find new hyperparameters and report them

Furthermore, we are wondering if this modified ProgressCallback will work well in anomaly_detection and anomaly_segmentation

CI result: https://ci-ote.iotg.sclab.intel.com/job/ote/job/pr-ote-test/344/

[ashwinvaidya17]

Probably out of scope for this PR but it would be nice to have a separate field in TrainParameters for hpo instead of replacing the update_progress object. I have pushed the proposal here develop...av/refactor_proposal the changes here will have to be propagated to other tasks such as detection and segmentation. But the idea behind this is to start a discussion.

The diff looks big but the key lines are

1. develop...av/refactor_proposal#diff-5ebeab25c618a314b6c5aae1533f78bf52952aa8162d9a605b2722c6d26ab13eR38
2. develop...av/refactor_proposal#diff-b92b2e145585581d8620bb4a723a53fdfd3147e6f5315f9ecce049c153a9533aR1-R26
3. develop...av/refactor_proposal#diff-b92b2e145585581d8620bb4a723a53fdfd3147e6f5315f9ecce049c153a9533aR1-R26
4. develop...av/refactor_proposal#diff-29ea76f6a531718a62189344389ade91312b587b765314c2fb7af1be396abdf8R55
5. develop...av/refactor_proposal#diff-57f84e87ecf6bedddb537d000409e2d170ec5e1b513e96b5c554aca8e609cd1eR1-R42

Another idea is to have a list of callbacks in ote_sdk/ote_sdk/entities/train_parameters.py so that we can support more callbacks from ote side in the future.

@dataclass
class TrainParameters:

    resume: bool = False
    callbacks: List[Callback] = [] # from ote_sdk.usecases.reporting.callback import Callback
    save_model: Callable[[], None] = default_save_model_callback

There is no need to inherit anything in HPOCallback on the ote side.

class HpoCallback():
    """Callback class to report score to hpopt"""

    def __init__(self, hp_config, metric, hpo_task):
        super().__init__()
        self.hp_config = hp_config
        self.metric = metric
        self.hpo_task = hpo_task

    def __call__(self, score: Optional[float] = None):
        if score is not None:
            if hpopt.report(config=self.hp_config, score=score) == hpopt.Status.STOP:
                self.hpo_task.cancel_training()
    

[yunchu]

@ashwinvaidya17, @samet-akcay, if latest update looks good for you guys, please leave a comment. i think i can approve this PR.

[eunwoosh]

@ashwinvaidya17 I agree to your opinion. or maybe it's also better to be able to apply new callback as necessary. But you know, modification of ote_sdk is quite a big task. It would be nice to have conversation when planning next step of ote_sdk.

[ashwinvaidya17]

Looks fine to me but I'll wait for @samet-akcay's feedback

[samet-akcay]

Looks good, thanks a lot for the effort!

[samet-akcay]

@ashwinvaidya17's proposal would make it clearer on the ote side. of course, this would require some discussions and agreements before the implementation. @goodsong81, FYI

[samet-akcay]

Would be good to schedule a BMM build to ensure everything is fine on the UI side.

[yunchu]

run ote_sdk tests

[yunchu]

@samet-akcay if you mean the SC UI, it requires other changes to enable it for SC. will be addressed by other PR later.

[yunchu]

run ote-test

[eunwoosh]

run ote-test

[eunwoosh]

run ote-test

[goodsong81]

LGTM.
Failures are irrelevant to this change.