[ENH] defining boundary between sktime and pytorch-forecasting, support for foundation models

[fkiraly]

Issue collecting design discussion related to the boundary between sktime and pytorch-forecasting, with a particular focus on foundation models and weight management, also see sktime issue sktime/sktime#6177.

In sktime we have recently started to introduce a dedicated layer for pytorch based forecasting models, and also specifically around weight management for "foundation models" - pre-trained deep learning models, of transformer architecture, which are predominantly pytorch based as well.

This situation begs the natural question, whether some parts of this layer - and if yes, which precisely - would be better contained in pytorch-forecasting. For instance, one could argue that the natural boundaries of pytorch-forecasting are anything that has to do with torch objects and their direct interfaces, which would include the aforementioned foundation models.

One could even argue that all sktime interfaces specific to pytorch based forecasters should be contained in pytorch-forecasting, in the form of a 2nd party interface, along the lines of patterns discussed here: sktime/sktime#6639, that is, sktime estimators present and maintained in pytorch-forecasting towards with the pytorch-facing backend, only specific to forecasting. Although in a case where there are common backend concerns with, say, time series classification (FYI @fnhirwa), that might be cutting off too much.

[fkiraly]

Listing different layers and sublayers that might help to draw a delineation:

• forecasting framework layer, sktime BaseForecaster etc
• common pytorch framework layer objects, e.g., a hypothetical BasePytorchNetwork in sktime
• common pytorch framework layer objects specific to forecasting, e.g., a hypothetical BasePytorchForecaster in sktime
• concrete pytorch networks or layers that are specific to forecasting models, but not full forecasting models
• concrete pytorch networks or layers that are specific to time series models but shared by, say, forecasting and time series classification, see multimodal momentfm transformer modl [ENH] momentfm foundation models sktime/sktime#6542 (FYI @julian-fong)
• concrete pytorch forecasting models, e.g., nbeats forecaster