Merge branch 'master' into ETNA-1022-v2

.gitignore

@@ -130,6 +130,7 @@ checkpoints/
 poetry.toml
 
 *.html
+*.db
 .devcontainer
 /docs/source/api/
 tmp

CHANGELOG.md

@@ -11,13 +11,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Exogenous variables shift transform `ExogShiftTransform`([#1254](https://github.com/tinkoff-ai/etna/pull/1254))
 - Parameter `start_timestamp` to forecast CLI command ([#1265](https://github.com/tinkoff-ai/etna/pull/1265))
 - `DeepStateModel` ([#1253](https://github.com/tinkoff-ai/etna/pull/1253))
+- Function `estimate_max_n_folds` for folds number estimation ([#1279](https://github.com/tinkoff-ai/etna/pull/1279))
+- Parameters `estimate_n_folds` and `context_size` to forecast and backtest CLI commands ([#1284](https://github.com/tinkoff-ai/etna/pull/1284))
+-
+- Class `Tune` for hyperparameter optimization within existing pipeline ([#1200](https://github.com/tinkoff-ai/etna/pull/1200))
+- Add `etna.distributions` for using it instead of using `optuna.distributions` ([#1292](https://github.com/tinkoff-ai/etna/pull/1292))
+- 
+
 ### Changed
 - Set the default value of `final_model` to `LinearRegression(positive=True)` in the constructor of `StackingEnsemble` ([#1238](https://github.com/tinkoff-ai/etna/pull/1238))
 - Add microseconds to `FileLogger`'s directory name ([#1264](https://github.com/tinkoff-ai/etna/pull/1264))
 - Inherit `SaveMixin` from `AbstractSaveable` for mypy checker ([#1261](https://github.com/tinkoff-ai/etna/pull/1261))
 - Update requirements for `holidays` and `scipy`, change saving library from `pickle` to `dill` in `SaveMixin` ([#1268](https://github.com/tinkoff-ai/etna/pull/1268))
 - Update requirement for `ruptures`, add requirement for `sqlalchemy` ([#1276](https://github.com/tinkoff-ai/etna/pull/1276))
--
+- Optimize `make_samples` of `RNNNet` and `MLPNet` ([#1281](https://github.com/tinkoff-ai/etna/pull/1281))
+- Remove `to_be_fixed` from inference tests on `SpecialDaysTransform` ([#1283](https://github.com/tinkoff-ai/etna/pull/1283))
+- Rewrite `TimeSeriesImputerTransform` to work without per-segment wrapper ([#1293](https://github.com/tinkoff-ai/etna/pull/1293))
+- 
+- 
+- Add default `params_to_tune` for catboost models ([#1185](https://github.com/tinkoff-ai/etna/pull/1185))
+- Add default `params_to_tune` for `ProphetModel` ([#1203](https://github.com/tinkoff-ai/etna/pull/1203))
+- Add default `params_to_tune` for `SARIMAXModel`, change default parameters for the model ([#1206](https://github.com/tinkoff-ai/etna/pull/1206))
+- Add default `params_to_tune` for linear models ([#1204](https://github.com/tinkoff-ai/etna/pull/1204))
+- Add default `params_to_tune` for `SeasonalMovingAverageModel`, `MovingAverageModel`, `NaiveModel` and `DeadlineMovingAverageModel` ([#1208](https://github.com/tinkoff-ai/etna/pull/1208))
+- Add default `params_to_tune` for `DeepARModel` and `TFTModel` ([#1210](https://github.com/tinkoff-ai/etna/pull/1210))
+- Add default `params_to_tune` for `HoltWintersModel`, `HoltModel` and `SimpleExpSmoothingModel` ([#1209](https://github.com/tinkoff-ai/etna/pull/1209))
+- Add default `params_to_tune` for `RNNModel` and `MLPModel` ([#1218](https://github.com/tinkoff-ai/etna/pull/1218))
+- Add default `params_to_tune` for `DateFlagsTransform`, `TimeFlagsTransform`, `SpecialDaysTransform` and `FourierTransform` ([#1228](https://github.com/tinkoff-ai/etna/pull/1228))
+- Add default `params_to_tune` for `MedianOutliersTransform`, `DensityOutliersTransform` and `PredictionIntervalOutliersTransform` ([#1231](https://github.com/tinkoff-ai/etna/pull/1231))
+- Add default `params_to_tune` for `TimeSeriesImputerTransform` ([#1232](https://github.com/tinkoff-ai/etna/pull/1232))
+- Add default `params_to_tune` for `DifferencingTransform`, `MedianTransform`, `MaxTransform`, `MinTransform`, `QuantileTransform`, `StdTransform`, `MeanTransform`, `MADTransform`, `MinMaxDifferenceTransform`, `SumTransform`, `BoxCoxTransform`, `YeoJohnsonTransform`, `MaxAbsScalerTransform`, `MinMaxScalerTransform`, `RobustScalerTransform` and `StandardScalerTransform` ([#1233](https://github.com/tinkoff-ai/etna/pull/1233))
+- Add default `params_to_tune` for `LabelEncoderTransform` ([#1242](https://github.com/tinkoff-ai/etna/pull/1242))
+- Add default `params_to_tune` for `ChangePointsSegmentationTransform`, `ChangePointsTrendTransform`, `ChangePointsLevelTransform`, `TrendTransform`, `LinearTrendTransform`, `TheilSenTrendTransform` and `STLTransform` ([#1243](https://github.com/tinkoff-ai/etna/pull/1243))
+- Add default `params_to_tune` for `TreeFeatureSelectionTransform`, `MRMRFeatureSelectionTransform` and `GaleShapleyFeatureSelectionTransform` ([#1250](https://github.com/tinkoff-ai/etna/pull/1250))
+- Add tuning stage into `Auto.fit` ([#1272](https://github.com/tinkoff-ai/etna/pull/1272))
+- Add `params_to_tune` into `Tune` init ([#1282](https://github.com/tinkoff-ai/etna/pull/1282))
+- Skip duplicates during `Tune.fit`, skip duplicates in `top_k`, add AutoML notebook ([#1285](https://github.com/tinkoff-ai/etna/pull/1285))
 ### Fixed
 - Fix `plot_backtest` and `plot_backtest_interactive` on one-step forecast ([1260](https://github.com/tinkoff-ai/etna/pull/1260))
 - Fix `BaseReconciliator` to work on `pandas==1.1.5` ([#1229](https://github.com/tinkoff-ai/etna/pull/1229))
@@ -30,6 +59,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Fix problems with flake8 B023 ([#1252](https://github.com/tinkoff-ai/etna/pull/1252))
 - Fix problem with swapped forecast methods in HierarchicalPipeline ([#1259](https://github.com/tinkoff-ai/etna/pull/1259))
 - Fix problem with segment name "target" in `StackingEnsemble` ([#1262](https://github.com/tinkoff-ai/etna/pull/1262))
+- Fix `BasePipeline.forecast` when prediction intervals are estimated on history data with presence of NaNs ([#1291](https://github.com/tinkoff-ai/etna/pull/1291))
+- Teach `BaseMixin.set_params` to work with nested `list` and `tuple` ([#1201](https://github.com/tinkoff-ai/etna/pull/1201))
 
 ## [2.0.0] - 2023-04-11
 ### Added

README.md

@@ -139,9 +139,11 @@ pip install etna
 
 The default version doesn't contain all the dependencies, because some of them are needed only for specific models, e.g. Prophet, PyTorch.
 Available user extensions are the following:
-* `prophet`
-* `torch`
-* `wandb`
+* `prophet`: adds prophet model`,
+* `torch`: adds models based on neural nets,
+* `wandb`: adds wandb logger,
+* `auto`: adds AutoML functionality,
+* `classiciation`: adds time series classification functionality.
 
 Install extension:
 ```bash

docs/source/api.rst

@@ -17,6 +17,7 @@ API
    reconciliation
    analysis
    auto
+   distributions
    clustering
    loggers
    commands

docs/source/commands.rst

@@ -23,14 +23,20 @@ Basic ``forecast`` usage:
 
 **Forecast config parameters**
 
-* :code:`prediction_interval` - whether to estimate prediction interval for forecast.
-* :code:`quantiles` - levels of prediction distribution. By default 2.5% and 97.5% are taken to form a 95% prediction interval.
-* :code:`n_folds` - number of folds to use in the backtest for prediction interval estimation. By default equals to 3.
-* :code:`return_components` - whether to estimate forecast components
 * :code:`start_timestamp` - timestamp with the starting point of forecast.
+* :code:`estimate_n_folds` - whether to estimate the number of folds from data. Works only when prediction intervals are enabled. Requires :code:`context_size` parameter set in pipeline config.
+
+Other parameters that could be set in the configuration file could be found in :meth:`~etna.pipeline.pipeline.Pipeline.forecast` method documentation.
 
 Setting these parameters is optional.
-Further information on arguments could be found in the documentation of :meth:`~etna.pipeline.pipeline.Pipeline.forecast` method.
+
+
+**Pipeline config parameters**
+
+* :code:`context_size` - minimum number of points in the history that is required by pipeline to produce a forecast.
+
+Further information on pipeline parameters could be found in :class:`~etna.pipeline.pipeline.Pipeline` method documentation.
+
 
 **How to create config?**
 
@@ -63,6 +69,26 @@ Parameter :code:`start_timestamp` could be set similarly:
     quantiles: [0.025, 0.975]
     start_timestamp: "2020-01-12"
 
+Example of a pair of configs for number of folds estimation:
+
+.. code-block:: yaml
+
+    _target_: etna.pipeline.Pipeline
+    horizon: 4
+    model:
+      _target_: etna.models.CatBoostMultiSegmentModel
+    transforms:
+      - _target_: etna.transforms.LinearTrendTransform
+        in_column: target
+      - _target_: etna.transforms.SegmentEncoderTransform
+    context_size: 1
+
+.. code-block:: yaml
+
+    prediction_interval: true
+    quantiles: [0.025, 0.975]
+    estimate_n_folds: true
+
 **How to prepare data?**
 
 Example of dataset with data to forecast:
@@ -114,6 +140,20 @@ Basic ``backtest`` usage:
             [EXOG_PATH]             path to csv with exog data
             [KNOWN_FUTURE]          list of all known_future columns (regressor columns). If not specified then all exog_columns considered known_future [default: None]
 
+**Backtest config parameters**
+
+* :code:`estimate_n_folds` - whether to estimate the number of folds from data. Requires :code:`context_size` parameter set in pipeline config.
+
+Other parameters that could be set in the configuration file could be found in :meth:`~etna.pipeline.base.BasePipeline.backtest` method documentation.
+
+Setting these parameters is optional.
+
+
+**Pipeline config parameters**
+
+* :code:`context_size` - minimum number of points in the history that is required by pipeline to produce a forecast.
+
+Further information on pipeline parameters could be found in :class:`~etna.pipeline.pipeline.Pipeline` method documentation.
 
 **How to create configs?**
 
@@ -142,6 +182,29 @@ Example of backtest run config:
       - _target_: etna.metrics.MAPE
       - _target_: etna.metrics.SMAPE
 
+Example of a pair of configs for number of folds estimation for backtest:
+
+.. code-block:: yaml
+
+    _target_: etna.pipeline.Pipeline
+    horizon: 4
+    model:
+      _target_: etna.models.CatBoostMultiSegmentModel
+    transforms:
+      - _target_: etna.transforms.LinearTrendTransform
+        in_column: target
+      - _target_: etna.transforms.SegmentEncoderTransform
+    context_size: 1
+
+.. code-block:: yaml
+
+    n_folds: 200
+    n_jobs: 4
+    metrics:
+      - _target_: etna.metrics.MAE
+      - _target_: etna.metrics.SMAPE
+    estimate_n_folds: true
+
 
 **How to prepare data?**
 

docs/source/conf.py

@@ -73,7 +73,7 @@
     "scipy": ("https://docs.scipy.org/doc/scipy/", None),
     "torch": ("https://pytorch.org/docs/stable/", None),
     "pytorch_lightning": ("https://pytorch-lightning.readthedocs.io/en/stable/", None),
-    "optuna": ("https://optuna.readthedocs.io/en/stable/", None)
+    "optuna": ("https://optuna.readthedocs.io/en/v2.10.1/", None)
 }
 
 autodoc_typehints = "both"

etna/auto/__init__.py

@@ -1,2 +1,3 @@
 from etna.auto.auto import Auto
+from etna.auto.auto import Tune
 from etna.auto.pool import Pool