Skip to content

Add refit parameter into backtest #1154

Closed
Mr-Geekman opened this issue Mar 6, 2023 · 1 comment
Closed

Add refit parameter into backtest #1154

Mr-Geekman opened this issue Mar 6, 2023 · 1 comment
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@Mr-Geekman
Copy link
Contributor

Mr-Geekman commented Mar 6, 2023

🚀 Feature Request

After adding possibility to forecast on future with gap, we can modify our backtest to work with this feature.

Proposal

Add parameter refit into BasePipeline.backtest.

Possible values:

  • True (default)
    • fit on every fold
    • all folds should be processed in parallel (like now)
  • False
    • fit only on the earliest fold
    • first fold is processed separately and pipeline is fit
    • consecutive folds should be processed in parallel
  • value: int
    • refit pipeline every value folds starting from the earliest
    • in every group of folds that uses the same pipeline we should work like with refit=False

How to make this change:

  • Extract logic of parallel run of folds from BasePipeline.backtest into a separate method
  • Modify BasePipeline._run_fold to return fitted pipeline and accept fitted pipeline
  • Implement logic with fitting folds and making predictions
    • We can interpret refit=True as refit: int = 1, because we consider each fold to be in the separate group
    • We can interpret refit=False as refit: int = n_folds, because we consider all folds to be in the same group

To discuss: what to do with FoldMask?

Test cases

  • Fix tests for changed methods
  • Check that n_jobs doesn't influence the results with different refit
  • Check that pipeline is working with models and transforms that supports inference mode
  • Check that pipeline doesn't work with models and transforms that doesn't support inference mode

Additional context

No response

@Mr-Geekman Mr-Geekman added the enhancement New feature or request label Mar 6, 2023
@Mr-Geekman Mr-Geekman added this to the Inference 2.1 milestone Mar 6, 2023
@github-project-automation github-project-automation bot moved this to Specification in etna board Mar 6, 2023
@Mr-Geekman Mr-Geekman self-assigned this Mar 7, 2023
@Mr-Geekman Mr-Geekman moved this from Specification to In Progress in etna board Mar 7, 2023
@Mr-Geekman Mr-Geekman moved this from In Progress to In Review in etna board Mar 9, 2023
@Mr-Geekman
Copy link
Contributor Author

Closed by #1159.

@github-project-automation github-project-automation bot moved this from In Review to Done in etna board Mar 14, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
Status: Done
Development

No branches or pull requests

1 participant