mlr3hyperband

Package website: release | dev

mlr3hyperband adds the optimization algorithms Successive Halving (Jamieson and Talwalkar 2016) and Hyperband (Li et al. 2018) to the mlr3 ecosystem. The implementation in mlr3hyperband features improved scheduling and parallelizes the evaluation of configurations. The package includes tuners for hyperparameter optimization in mlr3tuning and optimizers for black-box optimization in bbotk.

Resources

There are several sections about hyperparameter optimization in the mlr3book.

The gallery features a series of case studies on Hyperband.

• Tune the hyperparameters of XGBoost with Hyperband
• Use data subsampling and Hyperband to optimize a support vector machine.

Installation

Install the last release from CRAN:

install.packages("mlr3hyperband")

Install the development version from GitHub:

remotes::install_github("mlr-org/mlr3hyperband")

Examples

We optimize the hyperparameters of an XGBoost model on the Sonar data set. The number of boosting rounds nrounds is the fidelity parameter. We tag this parameter with "budget" in the search space.

library(mlr3hyperband)
library(mlr3learners)

learner = lrn("classif.xgboost",
  nrounds           = to_tune(p_int(27, 243, tags = "budget")),
  eta               = to_tune(1e-4, 1, logscale = TRUE),
  max_depth         = to_tune(1, 20),
  colsample_bytree  = to_tune(1e-1, 1),
  colsample_bylevel = to_tune(1e-1, 1),
  lambda            = to_tune(1e-3, 1e3, logscale = TRUE),
  alpha             = to_tune(1e-3, 1e3, logscale = TRUE),
  subsample         = to_tune(1e-1, 1)
)

We use the tune() function to run the optimization.

instance = tune(
  tnr("hyperband", eta = 3),
  task = tsk("pima"),
  learner = learner,
  resampling = rsmp("cv", folds = 3),
  measures = msr("classif.ce")
)

The instance contains the best-performing hyperparameter configuration.

instance$result

##    nrounds       eta max_depth colsample_bytree colsample_bylevel    lambda     alpha subsample
## 1:      27 -2.102951         3        0.7175178         0.5419011 -5.390012 -4.696385  0.193622
## 3 variables not shown: [learner_param_vals, x_domain, classif.ce]

The archive contains all evaluated hyperparameter configurations. Hyperband adds the "stage" and "braket".

as.data.table(instance$archive)[, .(stage, bracket, classif.ce, nrounds)]

##     stage bracket classif.ce nrounds
##  1:     0       2  0.3489583      27
##  2:     0       2  0.2434896      27
##  3:     0       2  0.2591146      27
##  4:     0       2  0.3489583      27
##  5:     0       2  0.5052083      27
## ---                                 
## 18:     0       0  0.2434896     243
## 19:     0       0  0.4960938     243
## 20:     0       0  0.2903646     243
## 21:     2       2  0.2473958     243
## 22:     1       1  0.2421875     243

We fit a final model with optimized hyperparameters to make predictions on new data.

learner$param_set$values = instance$result_learner_param_vals
learner$train(tsk("sonar"))

References

Jamieson, Kevin, and Ameet Talwalkar. 2016. “Non-Stochastic Best Arm Identification and Hyperparameter Optimization.” In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, edited by Arthur Gretton and Christian C. Robert, 51:240–48. Proceedings of Machine Learning Research. Cadiz, Spain: PMLR. http://proceedings.mlr.press/v51/jamieson16.html.

Li, Lisha, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2018. “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization.” Journal of Machine Learning Research 18 (185): 1–52. https://jmlr.org/papers/v18/16-558.html.