How to change evaluation interval

[AhmedThahir]

I would like to perform evaluation of metrics in train and validation set only every train_every and val_every epochs, not every epoch.

Rather than changing the callback, is there an easier way to change this behavior?

I am looking for something like this:
https://lightning.ai/docs/pytorch/stable/common/trainer.html#check-val-every-n-epoch

[ottonemo]

I think this can be easily achieved using a custom scoring callback that invokes the scoring you want or returns np.nan. Just make sure that whatever you use to evaluate the logged metrics can handle np.nan.

Example:

import numpy as np
from sklearn.mertrics import accuracy_score
import skorch

def my_score(net, X, y):
    if len(net.history) % 10 != 0:
        return np.nan
    return accuracy_score(y, net.predict(X))

net = skorch.NeuralNetClassifier(
    mymod,
    callbacks=[
        EpochScoring(my_score),
    ],
)

Does this help?

[AhmedThahir]

Will this disable the default evaluations of train and validation loss?

Context: I do not want to keep evaluating every epoch to save up on resources.

[ottonemo]

No, I assumed (wrongly, as it seems), that you have an additional score that is costly and you want to reduce cost here.

In that case, no, you'd have to overwrite the callback:

def my_val_score(net, X, y):
    if len(net.history) % 5 != 0:
        return np.nan
    return net.get_loss(net.forward(X, device=net.device, training=False), y, X=X).item()

net = skorch.NeuralNetClassifier(
    mymod,
    callbacks__valid_loss=skorch.callbacks.EpochScoring(my_val_score, name='valid_loss'),
)

[AhmedThahir]

Thank you!