2.5 Resampling

Resampling strategies are usually used to assess the performance of a learning algorithm. mlr3 entails 6 predefined resampling strategies: Cross-validation, Leave-one-out cross validation, Repeated cross-validation, Out-of-bag bootstrap and other variants (e.g. b632), Monte-Carlo cross-validation and Holdout. The following sections provide guidance on how to set and select a resampling strategy and how to subsequently instantiate the resampling process.

2.5.1 Settings

In this example we use the iris task and a simple classification tree (package rpart).

task = tsk("iris")
learner = lrn("classif.rpart")

When performing resampling with a dataset, we first need to define which approach should be used. The resampling strategies of mlr3 can be queried using the .$keys() method of the mlr_resamplings dictionary.

mlr_resamplings
## <DictionaryResampling> with 6 stored values
## Keys: bootstrap, custom, cv, holdout, repeated_cv, subsampling

Additional resampling methods for special use cases will be available via extension packages, such as mlr3spatiotemporal for spatial data (still in development).

The model fit conducted in the train/predict/score chapter is equivalent to a “holdout”, so let’s consider this one first. Again, we can retrieve elements via $get() or with a convenience function (rsmp()):

resampling = rsmp("holdout")
print(resampling)
## <ResamplingHoldout> with 1 iterations
## * Instantiated: FALSE
## * Parameters: ratio=0.6667

Note that the Instantiated field is set to FALSE. This means we did not actually apply the strategy on a dataset yet, but just performed a dry-run. Applying the strategy on a dataset is done in section next Instantiation.

By default we get a .66/.33 split of the data. There are two ways in which the ratio can be changed:

Overwriting the slot in .$param_set$values using a named list:

resampling$param_set$values = list(ratio = 0.8)

Specifying the resampling parameters directly during construction:

rsmp("holdout", ratio = 0.8)
## <ResamplingHoldout> with 1 iterations
## * Instantiated: FALSE
## * Parameters: ratio=0.8

2.5.2 Instantiation

So far we just set the stage and selected the resampling strategy. To actually perform the splitting, we need to apply the settings on a dataset. This can be done in two ways:

Manually by calling the method .$instantiate() on a Task:

resampling = rsmp("cv", folds = 3L)
resampling$instantiate(task)
resampling$iters
## [1] 3
resampling$train_set(1)
##   [1]   3   4   6  13  22  25  26  28  29  34  38  39  42  45  48  52  58  59
##  [19]  65  66  71  73  74  78  80  82  83  84  87  90  96  97  99 104 108 110
##  [37] 111 112 115 116 118 121 124 128 131 133 134 140 142 147   1   2   5   7
##  [55]   9  11  12  14  16  18  19  27  30  33  37  40  43  46  50  54  55  56
##  [73]  62  63  69  70  75  85  86  91  93  94  95  98 100 102 105 106 107 113
##  [91] 114 117 120 129 132 137 139 144 146 150

Automatically by passing the resampling object to resample(). Here, the splitting is done within the resample() call based on the supplied Task:

learner1 = lrn("classif.rpart") # simple classification tree
learner2 = lrn("classif.featureless") # featureless learner, prediction majority class
rr1 = resample(task, learner1, resampling)
rr2 = resample(task, learner2, resampling)

setequal(rr1$resampling$train_set(1), rr2$resampling$train_set(1))
## [1] TRUE

If you want to compare multiple learners, you should use the same resampling per task to reduce the variance of the performance estimation (method 1). If you use method 2 (and do not instantiate manually before), the resampling splits will differ between both runs.

If your aim is to compare different Task, Learner or Resampling, you are better off using the benchmark() function. It is basically a wrapper around resample() simplifying the handling of multiple settings.

If you discover this only after you’ve run multiple resample() calls, don’t worry. You can transform multiple single ResampleResult objects into a BenchmarkResult (explained later) using the .$combine() method.

2.5.3 Execution

With a Task, a Learner and Resampling object we can call resample() and create a ResampleResult object.

Before we go into more detail, let’s change the resampling to a “3-fold cross-validation” to better illustrate what operations are possible with a ResampleResult. Additionally, we tell resample() to keep the fitted models via the flag store_models:

task = tsk("iris")
learner = lrn("classif.rpart")
resampling = rsmp("cv", folds = 3L)

rr = resample(task, learner, resampling, store_models = TRUE)
print(rr)
## <ResampleResult> of 3 iterations
## * Task: iris
## * Learner: classif.rpart
## * Warnings: 0 in 0 iterations
## * Errors: 0 in 0 iterations

The following operations are supported with ResampleResult objects:

Extract the performance for the individual resampling iterations:

rr$score(msr("classif.ce"))
##             task task_id               learner    learner_id     resampling
## 1: <TaskClassif>    iris <LearnerClassifRpart> classif.rpart <ResamplingCV>
## 2: <TaskClassif>    iris <LearnerClassifRpart> classif.rpart <ResamplingCV>
## 3: <TaskClassif>    iris <LearnerClassifRpart> classif.rpart <ResamplingCV>
##    resampling_id iteration prediction classif.ce
## 1:            cv         1     <list>       0.10
## 2:            cv         2     <list>       0.08
## 3:            cv         3     <list>       0.04

Extract and inspect the resampling splits:

rr$resampling
## <ResamplingCV> with 3 iterations
## * Instantiated: TRUE
## * Parameters: folds=3
rr$resampling$iters
## [1] 3
rr$resampling$test_set(1)
##  [1]   7   9  11  15  17  20  22  26  28  29  37  42  43  53  56  58  62  67  68
## [20]  70  73  77  78  85  86  87  88  89  94  95 101 102 103 105 106 107 108 113
## [39] 114 116 119 120 122 123 126 128 136 137 148 149
rr$resampling$train_set(3)
##   [1]   7   9  11  15  17  20  22  26  28  29  37  42  43  53  56  58  62  67
##  [19]  68  70  73  77  78  85  86  87  88  89  94  95 101 102 103 105 106 107
##  [37] 108 113 114 116 119 120 122 123 126 128 136 137 148 149   1   2   3   5
##  [55]  10  13  14  16  21  31  35  36  39  44  46  48  49  50  54  55  59  60
##  [73]  61  63  66  69  71  74  75  76  81  82  83  90  91  92  98  99 109 111
##  [91] 124 125 127 129 130 133 134 135 138 143

Retrieve the learner of a specific iteration and inspect it:

lrn = rr$learners[[1]]
lrn$model
## n= 100 
## 
## node), split, n, loss, yval, (yprob)
##       * denotes terminal node
## 
## 1) root 100 63 setosa (0.3700 0.3300 0.3000)  
##   2) Petal.Length< 2.45 37  0 setosa (1.0000 0.0000 0.0000) *
##   3) Petal.Length>=2.45 63 30 versicolor (0.0000 0.5238 0.4762)  
##     6) Petal.Length< 4.75 31  0 versicolor (0.0000 1.0000 0.0000) *
##     7) Petal.Length>=4.75 32  2 virginica (0.0000 0.0625 0.9375) *

2.5.4 Custom resampling

Sometimes it is necessary to perform resampling with custom splits. If you want to do that because you are coming from a specific modeling field, first take a look at the mlr3 extension packages. It is important to make sure that your custom resampling method has not been implemented already.

If your custom resampling method is widely used in your field, feel welcome to integrate it into one of the existing mlr3 extension packages. You could also create your own extension package.

A manual resampling instance can be created using the "custom" template.

resampling = rsmp("custom")
resampling$instantiate(task,
  list(c(1:10, 51:60, 101:110)),
  list(c(11:20, 61:70, 111:120))
)
resampling$iters
## [1] 1
resampling$train_set(1)
##  [1]   1   2   3   4   5   6   7   8   9  10  51  52  53  54  55  56  57  58  59
## [20]  60 101 102 103 104 105 106 107 108 109 110
resampling$test_set(1)
##  [1]  11  12  13  14  15  16  17  18  19  20  61  62  63  64  65  66  67  68  69
## [20]  70 111 112 113 114 115 116 117 118 119 120