Feature: wrapper for callbacks

[ditwoo]

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contribution guide?
• Did you check the code style? catalyst-make-codestyle && catalyst-check-codestyle (pip install -U catalyst-codestyle).
• Did you make sure to update the docs? We use Google format for all the methods and classes.
• Did you check the docs with make check-docs?
• Did you write any new necessary tests?
• Did you add your new functionality to the docs?
• Did you update the CHANGELOG?
• You can use 'Login as guest' to see Teamcity build logs.

Description

Related Issue

Type of Change

• Examples / docs / tutorials / contributors update
• Bug fix (non-breaking change which fixes an issue)
• Improvement (non-breaking change which improves an existing feature)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

[Scitator]

just to be sure, could you please update docs with this new callbacks & changelog :)
I now it only draft yet :)

[Scitator]

speaking about architecture and implementation,
what do you think about CallbackFilterCallback?
not only with name filtering, but also with some condition one?

callback_filter = CallbackFilterCallback(
  key={"train": "metric-1"}, 
  lambda={"valid": lambda key, value: isinstance(value, MetricCallback)},
  # or, even we can also add something like
  value={"valid": MetricCallback}
)

long story shot, with callback filtering, most of the time you have 2 options:

1. filter by name (key field)
2. filter by class (value field)
3. some very custom condition -> lambda

what do you think about such proposal?

[Scitator]

moreover, if we would like to think about filtering in a broader sense....
why do we filter only on loader name? during stage, we have 2 fors - epoch and loader one.
In this case, I think, we could filter not only on loader name, but also on epoch index.

nevertheless, still now sure about user friendly API for this :)

[Scitator]

maybe, only maybe, something like

callback_filter = CallbackFilterCallback(
  condition=lambda epoch, loader: loader == "train",
  filter=lambda key, value: isinstance(value, MetricCallback),
)

[ditwoo]

@Scitator what do you think about FilterCallback or SkipCallback or FilterMetricCallback as a name for a class?
(I think that we can generate a better name without repeated callback word)

Also what do you think about use cases for arguments:

callback = FilterCallback(
    loader0="to_filter",  # base case - ignore callback for loader
    loader1=["to_fitler1", "to_filter2"],  # ignore multiple callbacks
    loader2={"to_filter1": [1, 22, 333], "to_filter2": [1, 2, 3]},  # turn off callbacks for epochs
    loader3=lambda epoch, loader_name, loader_obj: loader_obj,  # filter with custom function
)

[Scitator]

@ditwoo I think, FilterCallback would be good choice :)
First of all, we should not concentrate on Metric-based callbacks only.

[Scitator]

let's move this function to

catalyst/dl/utils/callbacks.py

so it would be easier to use it from anywhere

[Scitator]

and what do you think about splitting it into two separate callbacks, like

def get_original_callback(callback) -> callback:
  while not callback and is-wrapper:
    return callback.callback  # unwrap

def check_isinstance(first, second) -> bool:
  first = get_original_callback(first)
  second = get_original_callback(second)
  output = isinstance(first, second)
  return output

[Scitator]

btw, could you please move your tests to .py files,
like https://github.com/catalyst-team/catalyst/blob/master/catalyst/contrib/dl/callbacks/tests/test_tracer_callback.py
?

[Scitator]

🤔 what do you think about full-featured examples?
we have really tiny one:

import torch
from torch.utils.data import DataLoader, TensorDataset
from catalyst.dl import SupervisedRunner

num_samples, num_features = int(1e4), int(1e1)
X, y = torch.rand(num_samples, num_features), torch.rand(num_samples)
loader = DataLoader(TensorDataset(X, y), batch_size=32, num_workers=1)
loaders = {"train": loader, "valid": loader}

model = torch.nn.Linear(num_features, 1)
criterion = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters())
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, [3, 6])

runner = SupervisedRunner()
runner.train(
    model=model, criterion=criterion, optimizer=optimizer, scheduler=scheduler,
    loaders=loaders, logdir="./logdir", num_epochs=8, verbose=True,
)

[Scitator]

check_callback_isinstance?

[Scitator]

🤔 based on provided example, should we split it's functionality to

loaders=[] # loader names, where callback should be executed
ignore_loaders=[] # loader names, where we should ignore the callback

?

just wondering, how it would be more user-friendly

[Scitator]

btw, it also looks like this examples should not work because by default our metric - "loss" on "valid" loader, that is not computed here :)

[Scitator]

I think, we could easily use dl.AccuracyCallback instead

[Scitator]

maybe, we should move it to catalyst/core/callback?

[ditwoo]

sure