5.2 Resampling

Now, we can pass all created objects to the resample() function to get an object of class ResampleResult:

rr = resample(task, learner, resampling)
## INFO [mlr3] Running learner 'classif.rpart' on task 'iris' (iteration 1/1)'
print(rr)
## <ResampleResult> of learner 'iris' on task 'classif.rpart' with 1 iterations
##     Measure Min. 1st Qu. Median Mean 3rd Qu. Max. Sd
##  classif.ce  0.1     0.1    0.1  0.1     0.1  0.1 NA

Before we go into more detail, let’s change the resampling to a 3-fold cross-validation to better illustrate what operations are possible with a resampling result.

resampling = mlr_resamplings$get("cv", param_vals = list(folds = 3L))
rr = resample(task, learner, resampling)
## INFO [mlr3] Running learner 'classif.rpart' on task 'iris' (iteration 1/3)'
## INFO [mlr3] Running learner 'classif.rpart' on task 'iris' (iteration 2/3)'
## INFO [mlr3] Running learner 'classif.rpart' on task 'iris' (iteration 3/3)'
print(rr)
## <ResampleResult> of learner 'iris' on task 'classif.rpart' with 3 iterations
##     Measure Min. 1st Qu. Median Mean 3rd Qu. Max.   Sd
##  classif.ce 0.04    0.05   0.06 0.06    0.07 0.08 0.02

We can do different things with resampling results, e.g.:

  • Extract the performance for the individual resampling iterations:
rr$performance("classif.ce")
## [1] 0.08 0.04 0.06
  • Extract and inspect the now created resampling:
rr$resampling
## <ResamplingCV> with 3 iterations
## Instantiated: TRUE
## Parameters: folds=3
## 
## Public: clone, duplicated_ids, format, hash,
##   id, instance, instantiate, is_instantiated,
##   iters, param_set, task_hash, test_set,
##   train_set
rr$resampling$iters
## [1] 3
rr$resampling$test_set(1)
##  [1]   2   3   5   8   9  10  13  14  15  17  19  20
## [13]  23  25  29  30  34  35  43  45  46  48  51  52
## [25]  53  54  56  60  64  71  78  79  82  84  91  99
## [37] 100 105 108 114 118 119 121 122 130 134 138 143
## [49] 144 150
rr$resampling$test_set(2)
##  [1]   4  12  18  27  28  32  36  37  38  39  40  41
## [13]  47  57  62  63  66  74  75  80  90  92  93  94
## [25]  95  96  97 101 102 103 106 109 110 111 112 115
## [37] 117 120 124 125 126 128 129 132 133 139 140 147
## [49] 148 149
rr$resampling$test_set(3)
##  [1]   1   6   7  11  16  21  22  24  26  31  33  42
## [13]  44  49  50  55  58  59  61  65  67  68  69  70
## [25]  72  73  76  77  81  83  85  86  87  88  89  98
## [37] 104 107 113 116 123 127 131 135 136 137 141 142
## [49] 145 146
  • Retrieve the experiment of a specific iteration and inspect it:
e = rr$experiment(iter = 1)
e$model
## n= 100 
## 
## node), split, n, loss, yval, (yprob)
##       * denotes terminal node
## 
## 1) root 100 63 virginica (0.28000 0.35000 0.37000)  
##   2) Petal.Width< 1.65 65 30 versicolor (0.43077 0.53846 0.03077)  
##     4) Petal.Length< 2.5 28  0 setosa (1.00000 0.00000 0.00000) *
##     5) Petal.Length>=2.5 37  2 versicolor (0.00000 0.94595 0.05405) *
##   3) Petal.Width>=1.65 35  0 virginica (0.00000 0.00000 1.00000) *