## 3.1 Hyperparameter Tuning

Hyperparameters are second-order parameters of machine learning models that, while often not explicitly optimized during the model estimation process, can have important impacts on the outcome and predictive performance of a model. Typically, hyperparameters are fixed before training a model. However, because the output of a model can be sensitive to the specification of hyperparameters, it is often recommended to make an informed decision about which hyperparameter settings may yield better model performance. In many cases, hyperparameter settings may be chosen a priori, but it can be advantageous to try different settings before fitting your model on the training data. This process is often called ‘tuning’ your model.

Hyperparameter tuning is supported via the extension package mlr3tuning. Below you can find an illustration of the process:

At the heart of mlr3tuning are the R6 classes:

### 3.1.1 The TuningInstance Classes

The following sub-section examines the optimization of a simple classification tree on the Pima Indian Diabetes data set.

task = tsk("pima")
print(task)
## <TaskClassif:pima> (768 x 9)
## * Target: diabetes
## * Properties: twoclass
## * Features (8):
##   - dbl (8): age, glucose, insulin, mass, pedigree, pregnant, pressure,
##     triceps

We use the classification tree from rpart and choose a subset of the hyperparameters we want to tune. This is often referred to as the “tuning space”.

learner = lrn("classif.rpart")
learner$param_set ## <ParamSet> ## id class lower upper levels default value ## 1: minsplit ParamInt 1 Inf 20 ## 2: minbucket ParamInt 1 Inf <NoDefault[3]> ## 3: cp ParamDbl 0 1 0.01 ## 4: maxcompete ParamInt 0 Inf 4 ## 5: maxsurrogate ParamInt 0 Inf 5 ## 6: maxdepth ParamInt 1 30 30 ## 7: usesurrogate ParamInt 0 2 2 ## 8: surrogatestyle ParamInt 0 1 0 ## 9: xval ParamInt 0 Inf 10 0 ## 10: keep_model ParamLgl NA NA TRUE,FALSE FALSE Here, we opt to tune two parameters: • The complexity cp • The termination criterion minsplit The tuning space has to be bound, therefore one has to set lower and upper bounds: library("paradox") tune_ps = ParamSet$new(list(
ParamDbl$new("cp", lower = 0.001, upper = 0.1), ParamInt$new("minsplit", lower = 1, upper = 10)
))
tune_ps
## <ParamSet>
##          id    class lower upper levels        default value
## 1:       cp ParamDbl 0.001   0.1        <NoDefault[3]>
## 2: minsplit ParamInt 1.000  10.0        <NoDefault[3]>

Next, we need to specify how to evaluate the performance. For this, we need to choose a resampling strategy and a performance measure.

hout = rsmp("holdout")
measure = msr("classif.ce")

Finally, one has to select the budget available, to solve this tuning instance. This is done by selecting one of the available Terminators:

For this short introduction, we specify a budget of 20 evaluations and then put everything together into a TuningInstanceSingleCrit:

library("mlr3tuning")

evals20 = trm("evals", n_evals = 20)

instance = TuningInstanceSingleCrit$new( task = task, learner = learner, resampling = hout, measure = measure, search_space = tune_ps, terminator = evals20 ) instance ## <TuningInstanceSingleCrit> ## * State: Not optimized ## * Objective: <ObjectiveTuning:classif.rpart_on_pima> ## * Search Space: ## <ParamSet> ## id class lower upper levels default value ## 1: cp ParamDbl 0.001 0.1 <NoDefault[3]> ## 2: minsplit ParamInt 1.000 10.0 <NoDefault[3]> ## * Terminator: <TerminatorEvals> ## * Terminated: FALSE ## * Archive: ## <Archive> ## Null data.table (0 rows and 0 cols) To start the tuning, we still need to select how the optimization should take place. In other words, we need to choose the optimization algorithm via the Tuner class. ### 3.1.2 The Tuner Class The following algorithms are currently implemented in mlr3tuning: In this example, we will use a simple grid search with a grid resolution of 5. tuner = tnr("grid_search", resolution = 5) Since we have only numeric parameters, TunerGridSearch will create an equidistant grid between the respective upper and lower bounds. As we have two hyperparameters with a resolution of 5, the two-dimensional grid consists of $$5^2 = 25$$ configurations. Each configuration serves as hyperparameter setting for the previously defined Learner and triggers a 3-fold cross validation on the task. All configurations will be examined by the tuner (in a random order), until either all configurations are evaluated or the Terminator signals that the budget is exhausted. ### 3.1.3 Triggering the Tuning To start the tuning, we simply pass the TuningInstanceSingleCrit to the $optimize() method of the initialized Tuner. The tuner proceeds as follows:

1. The Tuner proposes at least one hyperparameter configuration (the Tuner may propose multiple points to improve parallelization, which can be controlled via the setting batch_size).
2. For each configuration, the given Learner is fitted on the Task using the provided Resampling. All evaluations are stored in the archive of the TuningInstanceSingleCrit.
3. The Terminator is queried if the budget is exhausted. If the budget is not exhausted, restart with 1) until it is.
4. Determine the configuration with the best observed performance.
5. Store the best configurations as result in the instance object. The best hyperparameter settings ($result_learner_param_vals) and the corresponding measured performance ($result_y) can be acessed from the instance.
tuner$optimize(instance) ## INFO [08:21:34.432] Starting to optimize 2 parameter(s) with '<OptimizerGridSearch>' and '<TerminatorEvals>' ## INFO [08:21:34.472] Evaluating 1 configuration(s) ## INFO [08:21:34.593] Result of batch 1: ## INFO [08:21:34.595] cp minsplit classif.ce resample_result ## INFO [08:21:34.595] 0.1 1 0.2461 <ResampleResult[18]> ## INFO [08:21:34.597] Evaluating 1 configuration(s) ## INFO [08:21:34.674] Result of batch 2: ## INFO [08:21:34.676] cp minsplit classif.ce resample_result ## INFO [08:21:34.676] 0.07525 5 0.2461 <ResampleResult[18]> ## INFO [08:21:34.678] Evaluating 1 configuration(s) ## INFO [08:21:34.721] Result of batch 3: ## INFO [08:21:34.722] cp minsplit classif.ce resample_result ## INFO [08:21:34.722] 0.07525 3 0.2461 <ResampleResult[18]> ## INFO [08:21:34.724] Evaluating 1 configuration(s) ## INFO [08:21:34.769] Result of batch 4: ## INFO [08:21:34.770] cp minsplit classif.ce resample_result ## INFO [08:21:34.770] 0.1 8 0.2461 <ResampleResult[18]> ## INFO [08:21:34.772] Evaluating 1 configuration(s) ## INFO [08:21:34.816] Result of batch 5: ## INFO [08:21:34.818] cp minsplit classif.ce resample_result ## INFO [08:21:34.818] 0.02575 10 0.2461 <ResampleResult[18]> ## INFO [08:21:34.820] Evaluating 1 configuration(s) ## INFO [08:21:34.870] Result of batch 6: ## INFO [08:21:34.872] cp minsplit classif.ce resample_result ## INFO [08:21:34.872] 0.001 8 0.2773 <ResampleResult[18]> ## INFO [08:21:34.873] Evaluating 1 configuration(s) ## INFO [08:21:34.922] Result of batch 7: ## INFO [08:21:34.924] cp minsplit classif.ce resample_result ## INFO [08:21:34.924] 0.07525 1 0.2461 <ResampleResult[18]> ## INFO [08:21:34.926] Evaluating 1 configuration(s) ## INFO [08:21:34.977] Result of batch 8: ## INFO [08:21:34.979] cp minsplit classif.ce resample_result ## INFO [08:21:34.979] 0.02575 5 0.2461 <ResampleResult[18]> ## INFO [08:21:34.981] Evaluating 1 configuration(s) ## INFO [08:21:35.037] Result of batch 9: ## INFO [08:21:35.039] cp minsplit classif.ce resample_result ## INFO [08:21:35.039] 0.1 10 0.2461 <ResampleResult[18]> ## INFO [08:21:35.041] Evaluating 1 configuration(s) ## INFO [08:21:35.089] Result of batch 10: ## INFO [08:21:35.091] cp minsplit classif.ce resample_result ## INFO [08:21:35.091] 0.07525 8 0.2461 <ResampleResult[18]> ## INFO [08:21:35.093] Evaluating 1 configuration(s) ## INFO [08:21:35.145] Result of batch 11: ## INFO [08:21:35.147] cp minsplit classif.ce resample_result ## INFO [08:21:35.147] 0.1 3 0.2461 <ResampleResult[18]> ## INFO [08:21:35.149] Evaluating 1 configuration(s) ## INFO [08:21:35.199] Result of batch 12: ## INFO [08:21:35.201] cp minsplit classif.ce resample_result ## INFO [08:21:35.201] 0.0505 1 0.2461 <ResampleResult[18]> ## INFO [08:21:35.203] Evaluating 1 configuration(s) ## INFO [08:21:35.253] Result of batch 13: ## INFO [08:21:35.255] cp minsplit classif.ce resample_result ## INFO [08:21:35.255] 0.0505 3 0.2461 <ResampleResult[18]> ## INFO [08:21:35.257] Evaluating 1 configuration(s) ## INFO [08:21:35.301] Result of batch 14: ## INFO [08:21:35.303] cp minsplit classif.ce resample_result ## INFO [08:21:35.303] 0.0505 10 0.2461 <ResampleResult[18]> ## INFO [08:21:35.305] Evaluating 1 configuration(s) ## INFO [08:21:35.357] Result of batch 15: ## INFO [08:21:35.358] cp minsplit classif.ce resample_result ## INFO [08:21:35.358] 0.001 3 0.2656 <ResampleResult[18]> ## INFO [08:21:35.360] Evaluating 1 configuration(s) ## INFO [08:21:35.415] Result of batch 16: ## INFO [08:21:35.416] cp minsplit classif.ce resample_result ## INFO [08:21:35.416] 0.001 5 0.2734 <ResampleResult[18]> ## INFO [08:21:35.418] Evaluating 1 configuration(s) ## INFO [08:21:35.465] Result of batch 17: ## INFO [08:21:35.467] cp minsplit classif.ce resample_result ## INFO [08:21:35.467] 0.02575 1 0.2461 <ResampleResult[18]> ## INFO [08:21:35.468] Evaluating 1 configuration(s) ## INFO [08:21:35.515] Result of batch 18: ## INFO [08:21:35.516] cp minsplit classif.ce resample_result ## INFO [08:21:35.516] 0.0505 8 0.2461 <ResampleResult[18]> ## INFO [08:21:35.518] Evaluating 1 configuration(s) ## INFO [08:21:35.565] Result of batch 19: ## INFO [08:21:35.566] cp minsplit classif.ce resample_result ## INFO [08:21:35.566] 0.07525 10 0.2461 <ResampleResult[18]> ## INFO [08:21:35.568] Evaluating 1 configuration(s) ## INFO [08:21:35.620] Result of batch 20: ## INFO [08:21:35.622] cp minsplit classif.ce resample_result ## INFO [08:21:35.622] 0.02575 3 0.2461 <ResampleResult[18]> ## INFO [08:21:35.627] Finished optimizing after 20 evaluation(s) ## INFO [08:21:35.628] Result: ## INFO [08:21:35.630] cp minsplit learner_param_vals x_domain classif.ce ## INFO [08:21:35.630] 0.1 1 <list[3]> <list[2]> 0.2461 ## cp minsplit learner_param_vals x_domain classif.ce ## 1: 0.1 1 <list[3]> <list[2]> 0.2461 instance$result_learner_param_vals
## $xval ## [1] 0 ## ##$cp
## [1] 0.1
##
## $minsplit ## [1] 1 instance$result_y
## classif.ce
##     0.2461

One can investigate all resamplings which were undertaken, as they are stored in the archive of the TuningInstanceSingleCrit and can be accessed through $data() method: instance$archive$data() ## cp minsplit classif.ce resample_result x_domain ## 1: 0.10000 1 0.2461 <ResampleResult[18]> <list[2]> ## 2: 0.07525 5 0.2461 <ResampleResult[18]> <list[2]> ## 3: 0.07525 3 0.2461 <ResampleResult[18]> <list[2]> ## 4: 0.10000 8 0.2461 <ResampleResult[18]> <list[2]> ## 5: 0.02575 10 0.2461 <ResampleResult[18]> <list[2]> ## 6: 0.00100 8 0.2773 <ResampleResult[18]> <list[2]> ## 7: 0.07525 1 0.2461 <ResampleResult[18]> <list[2]> ## 8: 0.02575 5 0.2461 <ResampleResult[18]> <list[2]> ## 9: 0.10000 10 0.2461 <ResampleResult[18]> <list[2]> ## 10: 0.07525 8 0.2461 <ResampleResult[18]> <list[2]> ## 11: 0.10000 3 0.2461 <ResampleResult[18]> <list[2]> ## 12: 0.05050 1 0.2461 <ResampleResult[18]> <list[2]> ## 13: 0.05050 3 0.2461 <ResampleResult[18]> <list[2]> ## 14: 0.05050 10 0.2461 <ResampleResult[18]> <list[2]> ## 15: 0.00100 3 0.2656 <ResampleResult[18]> <list[2]> ## 16: 0.00100 5 0.2734 <ResampleResult[18]> <list[2]> ## 17: 0.02575 1 0.2461 <ResampleResult[18]> <list[2]> ## 18: 0.05050 8 0.2461 <ResampleResult[18]> <list[2]> ## 19: 0.07525 10 0.2461 <ResampleResult[18]> <list[2]> ## 20: 0.02575 3 0.2461 <ResampleResult[18]> <list[2]> ## timestamp batch_nr ## 1: 2020-08-07 08:21:34 1 ## 2: 2020-08-07 08:21:34 2 ## 3: 2020-08-07 08:21:34 3 ## 4: 2020-08-07 08:21:34 4 ## 5: 2020-08-07 08:21:34 5 ## 6: 2020-08-07 08:21:34 6 ## 7: 2020-08-07 08:21:34 7 ## 8: 2020-08-07 08:21:34 8 ## 9: 2020-08-07 08:21:35 9 ## 10: 2020-08-07 08:21:35 10 ## 11: 2020-08-07 08:21:35 11 ## 12: 2020-08-07 08:21:35 12 ## 13: 2020-08-07 08:21:35 13 ## 14: 2020-08-07 08:21:35 14 ## 15: 2020-08-07 08:21:35 15 ## 16: 2020-08-07 08:21:35 16 ## 17: 2020-08-07 08:21:35 17 ## 18: 2020-08-07 08:21:35 18 ## 19: 2020-08-07 08:21:35 19 ## 20: 2020-08-07 08:21:35 20 In sum, the grid search evaluated 20/25 different configurations of the grid in a random order before the Terminator stopped the tuning. Now the optimized hyperparameters can take the previously created Learner, set the returned hyperparameters and train it on the full dataset. learner$param_set$values = instance$result_learner_param_vals
learner$train(task) The trained model can now be used to make a prediction on external data. Note that predicting on observations present in the task, should be avoided. The model has seen these observations already during tuning and therefore results would be statistically biased. Hence, the resulting performance measure would be over-optimistic. Instead, to get statistically unbiased performance estimates for the current task, nested resampling is required. ### 3.1.4 Automating the Tuning The AutoTuner wraps a learner and augments it with an automatic tuning for a given set of hyperparameters. Because the AutoTuner itself inherits from the Learner base class, it can be used like any other learner. Analogously to the previous subsection, a new classification tree learner is created. This classification tree learner automatically tunes the parameters cp and minsplit using an inner resampling (holdout). We create a terminator which allows 10 evaluations, and use a simple random search as tuning algorithm: library("paradox") library("mlr3tuning") learner = lrn("classif.rpart") tune_ps = ParamSet$new(list(
ParamDbl$new("cp", lower = 0.001, upper = 0.1), ParamInt$new("minsplit", lower = 1, upper = 10)
))
terminator = trm("evals", n_evals = 10)
tuner = tnr("random_search")

at = AutoTuner$new( learner = learner, resampling = rsmp("holdout"), measure = msr("classif.ce"), search_space = tune_ps, terminator = terminator, tuner = tuner ) at ## <AutoTuner:classif.rpart.tuned> ## * Model: - ## * Parameters: xval=0 ## * Packages: rpart ## * Predict Type: response ## * Feature types: logical, integer, numeric, factor, ordered ## * Properties: importance, missings, multiclass, selected_features, ## twoclass, weights We can now use the learner like any other learner, calling the $train() and $predict() method. This time however, we pass it to benchmark() to compare the tuner to a classification tree without tuning. This way, the AutoTuner will do its resampling for tuning on the training set of the respective split of the outer resampling. The learner then undertakes predictions using the test set of the outer resampling. This yields unbiased performance measures, as the observations in the test set have not been used during tuning or fitting of the respective learner. This is called nested resampling. To compare the tuned learner with the learner using its default, we can use benchmark(): grid = benchmark_grid( task = tsk("pima"), learner = list(at, lrn("classif.rpart")), resampling = rsmp("cv", folds = 3) ) # avoid console output from mlr3tuning logger = lgr::get_logger("bbotk") logger$set_threshold("warn")

bmr = benchmark(grid)
bmr\$aggregate(msrs(c("classif.ce", "time_train")))
##    nr      resample_result task_id          learner_id resampling_id iters
## 1:  1 <ResampleResult[18]>    pima classif.rpart.tuned            cv     3
## 2:  2 <ResampleResult[18]>    pima       classif.rpart            cv     3
##    classif.ce time_train
## 1:     0.2891     0.5667
## 2:     0.2591     0.0060

Note that we do not expect any differences compared to the non-tuned approach for multiple reasons:

• the task is too easy
• the task is rather small, and thus prone to overfitting
• the tuning budget (10 evaluations) is small
• rpart does not benefit that much from tuning

### References

Bergstra, James, and Yoshua Bengio. 2012. “Random Search for Hyper-Parameter Optimization.” J. Mach. Learn. Res. 13. JMLR.org: 281–305.