11  Nested Resampling

Evaluating a machine learning model often requires an additional layer of resampling when hyperparameters or features have to be selected. Nested resampling separates these model selection steps from the process estimating the performance of the model. If the same data is used for the model selection steps and the evaluation of the model itself, the resulting performance estimate of the model might be severely biased. One reason for this bias is that the repeated evaluation of the model on the test data could leak information about its structure into the model, this results in over-optimistic performance estimates. Keep in mind that nested resampling is a statistical procedure to estimate the predictive performance of the model trained on the full dataset. Nested resampling is not a procedure to select optimal hyperparameters. The resampling produces many hyperparameter configurations which should be not used to construct a final model (Simon 2007).

The graphic above illustrates nested resampling for hyperparameter tuning with 3-fold cross-validation in the outer resampling and 4-fold cross-validation in the inner resampling.

The nested resampling process:

  1. Uses a 3-fold cross-validation to get different testing and training data sets (outer resampling).
  2. Within the training data uses a 4-fold cross-validation to get different inner testing and training data sets (inner resampling).
  3. Tunes the hyperparameters using the inner data splits.
  4. Fits the learner on the outer training data set using the tuned hyperparameter configuration obtained with the inner resampling.
  5. Evaluates the performance of the learner on the outer testing data.
  6. 2-5 is repeated for each of the three folds (outer resampling).
  7. The three performance values are aggregated for an unbiased performance estimate.

See also this article for more explanations.

11.1 Execution

The previous section examined the optimization of a simple classification tree on the mlr_tasks_pima. We continue the example and estimate the predictive performance of the model with nested resampling.

We use a 4-fold cross-validation in the inner resampling loop. The AutoTuner executes the hyperparameter tuning and is stopped after 5 evaluations. The hyperparameter configurations are proposed by grid search.

library("mlr3verse")

learner = lrn("classif.rpart")
resampling = rsmp("cv", folds = 4)
measure = msr("classif.ce")
search_space = ps(cp = p_dbl(lower = 0.001, upper = 0.1))
terminator = trm("evals", n_evals = 5)
tuner = tnr("grid_search", resolution = 10)

at = AutoTuner$new(learner, resampling, measure, terminator, tuner, search_space)

A 3-fold cross-validation is used in the outer resampling loop. On each of the three outer train sets hyperparameter tuning is done and we receive three optimized hyperparameter configurations. To execute the nested resampling, we pass the AutoTuner to the resample() function. We have to set store_models = TRUE because we need the AutoTuner models to investigate the inner tuning.

task = tsk("pima")
outer_resampling = rsmp("cv", folds = 3)

rr = resample(task, at, outer_resampling, store_models = TRUE)
INFO  [21:33:31.204] [mlr3] Applying learner 'classif.rpart.tuned' on task 'pima' (iter 3/3) 
INFO  [21:33:31.275] [bbotk] Starting to optimize 1 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=5, k=0]' 
INFO  [21:33:31.337] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:31.353] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:31.361] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:31.394] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:31.418] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:31.485] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:31.512] [mlr3] Finished benchmark 
INFO  [21:33:31.556] [bbotk] Result of batch 1: 
INFO  [21:33:31.559] [bbotk]   cp classif.ce warnings errors runtime_learners 
INFO  [21:33:31.559] [bbotk]  0.1       0.25        0      0            0.101 
INFO  [21:33:31.559] [bbotk]                                 uhash 
INFO  [21:33:31.559] [bbotk]  464cc7eb-518f-40c2-956f-553e129bbae5 
INFO  [21:33:31.561] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:31.576] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:31.583] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:31.608] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:31.639] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:31.662] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:31.688] [mlr3] Finished benchmark 
INFO  [21:33:31.728] [bbotk] Result of batch 2: 
INFO  [21:33:31.731] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:31.731] [bbotk]  0.078  0.2441406        0      0            0.053 
INFO  [21:33:31.731] [bbotk]                                 uhash 
INFO  [21:33:31.731] [bbotk]  aabcd4ed-216f-43dc-ada9-5633b332cc89 
INFO  [21:33:31.732] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:31.747] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:31.760] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:31.784] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:31.807] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:31.830] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:31.858] [mlr3] Finished benchmark 
INFO  [21:33:31.906] [bbotk] Result of batch 3: 
INFO  [21:33:31.908] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:31.908] [bbotk]  0.001   0.234375        0      0            0.057 
INFO  [21:33:31.908] [bbotk]                                 uhash 
INFO  [21:33:31.908] [bbotk]  d91a476b-726e-41cc-995f-c21b623f85d8 
INFO  [21:33:31.910] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:31.924] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:31.931] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:31.957] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:31.982] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:32.008] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:32.041] [mlr3] Finished benchmark 
INFO  [21:33:32.082] [bbotk] Result of batch 4: 
INFO  [21:33:32.084] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:32.084] [bbotk]  0.089  0.2441406        0      0            0.066 
INFO  [21:33:32.084] [bbotk]                                 uhash 
INFO  [21:33:32.084] [bbotk]  588546c1-7f66-45d6-9f92-c112d256ce03 
INFO  [21:33:32.086] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:32.101] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:32.109] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:32.134] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:32.158] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:32.188] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:32.213] [mlr3] Finished benchmark 
INFO  [21:33:32.254] [bbotk] Result of batch 5: 
INFO  [21:33:32.256] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:32.256] [bbotk]  0.023  0.2285156        0      0            0.062 
INFO  [21:33:32.256] [bbotk]                                 uhash 
INFO  [21:33:32.256] [bbotk]  dea835a5-22a8-429f-888e-01b7326508da 
INFO  [21:33:32.264] [bbotk] Finished optimizing after 5 evaluation(s) 
INFO  [21:33:32.265] [bbotk] Result: 
INFO  [21:33:32.266] [bbotk]     cp learner_param_vals  x_domain classif.ce 
INFO  [21:33:32.266] [bbotk]  0.023          <list[2]> <list[1]>  0.2285156 
INFO  [21:33:32.315] [mlr3] Applying learner 'classif.rpart.tuned' on task 'pima' (iter 1/3) 
INFO  [21:33:32.363] [bbotk] Starting to optimize 1 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=5, k=0]' 
INFO  [21:33:32.366] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:32.380] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:32.388] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:32.419] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:32.442] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:32.466] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:32.492] [mlr3] Finished benchmark 
INFO  [21:33:32.531] [bbotk] Result of batch 1: 
INFO  [21:33:32.533] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:32.533] [bbotk]  0.067  0.2324219        0      0            0.053 
INFO  [21:33:32.533] [bbotk]                                 uhash 
INFO  [21:33:32.533] [bbotk]  683d9a82-e230-4342-b4f9-882544fa5e79 
INFO  [21:33:32.535] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:32.555] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:32.562] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:32.587] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:32.611] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:32.634] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:32.661] [mlr3] Finished benchmark 
INFO  [21:33:32.707] [bbotk] Result of batch 2: 
INFO  [21:33:32.709] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:32.709] [bbotk]  0.034  0.2324219        0      0            0.054 
INFO  [21:33:32.709] [bbotk]                                 uhash 
INFO  [21:33:32.709] [bbotk]  a867c4d8-7c3e-4832-a632-c6e465710cbd 
INFO  [21:33:32.711] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:32.726] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:32.733] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:32.758] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:32.783] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:32.813] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:32.838] [mlr3] Finished benchmark 
INFO  [21:33:32.881] [bbotk] Result of batch 3: 
INFO  [21:33:32.883] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:32.883] [bbotk]  0.012  0.2421875        0      0            0.061 
INFO  [21:33:32.883] [bbotk]                                 uhash 
INFO  [21:33:32.883] [bbotk]  9e36cc2d-ee8c-400b-9212-8e6e5cdc065f 
INFO  [21:33:32.885] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:32.900] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:32.908] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:32.938] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:32.962] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:32.985] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:33.011] [mlr3] Finished benchmark 
INFO  [21:33:33.059] [bbotk] Result of batch 4: 
INFO  [21:33:33.061] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:33.061] [bbotk]  0.056  0.2324219        0      0            0.054 
INFO  [21:33:33.061] [bbotk]                                 uhash 
INFO  [21:33:33.061] [bbotk]  70496280-e4f4-4b4f-a64a-408739009b32 
INFO  [21:33:33.063] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:33.077] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:33.085] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:33.109] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:33.133] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:33.157] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:33.190] [mlr3] Finished benchmark 
INFO  [21:33:33.236] [bbotk] Result of batch 5: 
INFO  [21:33:33.238] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:33.238] [bbotk]  0.078  0.2324219        0      0            0.054 
INFO  [21:33:33.238] [bbotk]                                 uhash 
INFO  [21:33:33.238] [bbotk]  054268f0-1079-439f-9166-18cd600c5909 
INFO  [21:33:33.245] [bbotk] Finished optimizing after 5 evaluation(s) 
INFO  [21:33:33.246] [bbotk] Result: 
INFO  [21:33:33.247] [bbotk]     cp learner_param_vals  x_domain classif.ce 
INFO  [21:33:33.247] [bbotk]  0.067          <list[2]> <list[1]>  0.2324219 
INFO  [21:33:33.292] [mlr3] Applying learner 'classif.rpart.tuned' on task 'pima' (iter 2/3) 
INFO  [21:33:33.346] [bbotk] Starting to optimize 1 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=5, k=0]' 
INFO  [21:33:33.349] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:33.364] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:33.371] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:33.397] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:33.427] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:33.451] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:33.478] [mlr3] Finished benchmark 
INFO  [21:33:33.517] [bbotk] Result of batch 1: 
INFO  [21:33:33.519] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:33.519] [bbotk]  0.012  0.2988281        0      0            0.062 
INFO  [21:33:33.519] [bbotk]                                 uhash 
INFO  [21:33:33.519] [bbotk]  2312427e-a8ae-406d-8379-d341599c56de 
INFO  [21:33:33.521] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:33.542] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:33.549] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:33.578] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:33.604] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:33.631] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:33.666] [mlr3] Finished benchmark 
INFO  [21:33:33.711] [bbotk] Result of batch 2: 
INFO  [21:33:33.713] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:33.713] [bbotk]  0.023  0.2988281        0      0            0.063 
INFO  [21:33:33.713] [bbotk]                                 uhash 
INFO  [21:33:33.713] [bbotk]  889d2751-6420-48ac-b278-7b5fc5af5173 
INFO  [21:33:33.715] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:33.731] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:33.740] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:33.768] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:33.802] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:33.827] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:33.854] [mlr3] Finished benchmark 
INFO  [21:33:33.899] [bbotk] Result of batch 3: 
INFO  [21:33:33.902] [bbotk]   cp classif.ce warnings errors runtime_learners 
INFO  [21:33:33.902] [bbotk]  0.1  0.2734375        0      0            0.065 
INFO  [21:33:33.902] [bbotk]                                 uhash 
INFO  [21:33:33.902] [bbotk]  129552ea-6a2c-4b6b-ae6c-9fe5f4e8c0e9 
INFO  [21:33:33.903] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:33.925] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:33.933] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:33.959] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:33.984] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:34.011] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:34.039] [mlr3] Finished benchmark 
INFO  [21:33:34.088] [bbotk] Result of batch 4: 
INFO  [21:33:34.090] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:34.090] [bbotk]  0.067  0.2734375        0      0            0.059 
INFO  [21:33:34.090] [bbotk]                                 uhash 
INFO  [21:33:34.090] [bbotk]  f3068ccb-8f6f-410c-9e41-f05ccb69ecbe 
INFO  [21:33:34.092] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:34.107] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:34.115] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:34.141] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:34.185] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:34.212] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:34.237] [mlr3] Finished benchmark 
INFO  [21:33:34.279] [bbotk] Result of batch 5: 
INFO  [21:33:34.281] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:34.281] [bbotk]  0.056  0.2734375        0      0             0.06 
INFO  [21:33:34.281] [bbotk]                                 uhash 
INFO  [21:33:34.281] [bbotk]  a8c54472-32a1-4785-986e-ade52f8b2fb5 
INFO  [21:33:34.288] [bbotk] Finished optimizing after 5 evaluation(s) 
INFO  [21:33:34.289] [bbotk] Result: 
INFO  [21:33:34.290] [bbotk]   cp learner_param_vals  x_domain classif.ce 
INFO  [21:33:34.290] [bbotk]  0.1          <list[2]> <list[1]>  0.2734375

You can freely combine different inner and outer resampling strategies. Nested resampling is not restricted to hyperparameter tuning. You can swap the AutoTuner for a AutoFSelector and estimate the performance of a model which is fitted on an optimized feature subset.

11.2 Evaluation

With the created ResampleResult we can now inspect the executed resampling iterations more closely. See the section on Resampling for more detailed information about ResampleResult objects.

We check the inner tuning results for stable hyperparameters. This means that the selected hyperparameters should not vary too much. We might observe unstable models in this example because the small data set and the low number of resampling iterations might introduces too much randomness. Usually, we aim for the selection of stable hyperparameters for all outer training sets.

   iteration    cp classif.ce learner_param_vals  x_domain task_id
1:         1 0.067  0.2324219          <list[2]> <list[1]>    pima
2:         2 0.100  0.2734375          <list[2]> <list[1]>    pima
3:         3 0.023  0.2285156          <list[2]> <list[1]>    pima
            learner_id resampling_id
1: classif.rpart.tuned            cv
2: classif.rpart.tuned            cv
3: classif.rpart.tuned            cv

Next, we want to compare the predictive performances estimated on the outer resampling to the inner resampling. Significantly lower predictive performances on the outer resampling indicate that the models with the optimized hyperparameters overfit the data.

rr$score()
                task task_id         learner          learner_id
1: <TaskClassif[50]>    pima <AutoTuner[42]> classif.rpart.tuned
2: <TaskClassif[50]>    pima <AutoTuner[42]> classif.rpart.tuned
3: <TaskClassif[50]>    pima <AutoTuner[42]> classif.rpart.tuned
           resampling resampling_id iteration              prediction
1: <ResamplingCV[20]>            cv         1 <PredictionClassif[20]>
2: <ResamplingCV[20]>            cv         2 <PredictionClassif[20]>
3: <ResamplingCV[20]>            cv         3 <PredictionClassif[20]>
   classif.ce
1:  0.2890625
2:  0.2500000
3:  0.2617188

The aggregated performance of all outer resampling iterations is essentially the unbiased performance of the model with optimal hyperparameter found by grid search.

rr$aggregate()
classif.ce 
 0.2669271

Note that nested resampling is computationally expensive. For this reason we use relatively small number of hyperparameter configurations and a low number of resampling iterations in this example. In practice, you normally have to increase both. As this is computationally intensive you might want to have a look at the section on Parallelization.

11.3 Final Model

We can use the AutoTuner to tune the hyperparameters of our learner and fit the final model on the full data set.

at$train(task)
INFO  [21:33:34.564] [bbotk] Starting to optimize 1 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=5, k=0]' 
INFO  [21:33:34.568] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:34.591] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:34.600] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:34.628] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:34.654] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:34.680] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:34.708] [mlr3] Finished benchmark 
INFO  [21:33:34.757] [bbotk] Result of batch 1: 
INFO  [21:33:34.759] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:34.759] [bbotk]  0.089   0.265625        0      0            0.059 
INFO  [21:33:34.759] [bbotk]                                 uhash 
INFO  [21:33:34.759] [bbotk]  0b287026-08a9-4216-a589-56d5aa58d341 
INFO  [21:33:34.761] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:34.777] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:34.785] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:34.812] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:34.839] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:34.867] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:34.904] [mlr3] Finished benchmark 
INFO  [21:33:34.944] [bbotk] Result of batch 2: 
INFO  [21:33:34.946] [bbotk]   cp classif.ce warnings errors runtime_learners 
INFO  [21:33:34.946] [bbotk]  0.1   0.265625        0      0            0.067 
INFO  [21:33:34.946] [bbotk]                                 uhash 
INFO  [21:33:34.946] [bbotk]  a5564e46-6f4e-4ad5-b649-422c61165df6 
INFO  [21:33:34.948] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:34.969] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:34.977] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:35.004] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:35.036] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:35.063] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:35.089] [mlr3] Finished benchmark 
INFO  [21:33:35.130] [bbotk] Result of batch 3: 
INFO  [21:33:35.132] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:35.132] [bbotk]  0.078  0.2513021        0      0            0.066 
INFO  [21:33:35.132] [bbotk]                                 uhash 
INFO  [21:33:35.132] [bbotk]  30a4aeda-8fe7-4647-b345-78d007e07e8c 
INFO  [21:33:35.134] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:35.156] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:35.165] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:35.190] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:35.214] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:35.238] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:35.264] [mlr3] Finished benchmark 
INFO  [21:33:35.311] [bbotk] Result of batch 4: 
INFO  [21:33:35.313] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:35.313] [bbotk]  0.045   0.234375        0      0            0.057 
INFO  [21:33:35.313] [bbotk]                                 uhash 
INFO  [21:33:35.313] [bbotk]  6eb6bee8-f0f0-4ffb-9eef-71ca94b0f1d0 
INFO  [21:33:35.315] [bbotk] Evaluating 1 configuration(s) 
INFO  [21:33:35.329] [mlr3] Running benchmark with 4 resampling iterations 
INFO  [21:33:35.336] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4) 
INFO  [21:33:35.361] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4) 
INFO  [21:33:35.385] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4) 
INFO  [21:33:35.417] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4) 
INFO  [21:33:35.444] [mlr3] Finished benchmark 
INFO  [21:33:35.487] [bbotk] Result of batch 5: 
INFO  [21:33:35.490] [bbotk]     cp classif.ce warnings errors runtime_learners 
INFO  [21:33:35.490] [bbotk]  0.001  0.2434896        0      0            0.059 
INFO  [21:33:35.490] [bbotk]                                 uhash 
INFO  [21:33:35.490] [bbotk]  2b83534e-9af8-4e08-af1e-9f152d61b4f2 
INFO  [21:33:35.496] [bbotk] Finished optimizing after 5 evaluation(s) 
INFO  [21:33:35.497] [bbotk] Result: 
INFO  [21:33:35.498] [bbotk]     cp learner_param_vals  x_domain classif.ce 
INFO  [21:33:35.498] [bbotk]  0.045          <list[2]> <list[1]>   0.234375

The trained model can now be used to make predictions on new data.

Warning

A common mistake is to report the performance estimated on the resampling sets on which the tuning was performed (at$tuning_result$classif.ce) as the model’s performance.

Instead, the performance estimated with nested resampling should be reported as the actual performance of the model.