3.6 Benchmarking

Comparing the performance of different learners on multiple tasks and/or different resampling schemes is a recurrent task. This operation is usually referred to as “benchmarking” in the field of machine-learning. mlr3 offers the benchmark() function for convenience.

3.6.1 Design Creation

In mlr3 we require you to supply a “design” of your benchmark experiment. By “design” we essentially mean the matrix of settings you want to execute. A “design” consists of Task, Learner and Resampling. Additionally, you can supply different Measure along side.

Here, we call benchmark() to perform a single holdout split on a single task and two learners. We use the expand_grid() function to create an exhaustive design and properly instantiate the resampling:

library(data.table)
design = expand_grid(
  tasks = mlr_tasks$mget("iris"),
  learners = mlr_learners$mget(c("classif.rpart", "classif.featureless")),
  resamplings = mlr_resamplings$mget("holdout")
)
print(design)
##             task                     learner          resampling
## 1: <TaskClassif>       <LearnerClassifRpart> <ResamplingHoldout>
## 2: <TaskClassif> <LearnerClassifFeatureless> <ResamplingHoldout>
bmr = benchmark(design)

Note that the holdout splits have been automatically instantiated for each row of the design. As a result, the rpart learner used a different training set than the featureless learner. However, for comparison of learners you usually want the learners to see the same splits into train and test sets. To overcome this issue, the resampling strategy needs to be manually instantiated before creating the design.

While the interface of benchmark() allows full flexibility, the creation of such design tables can be tedious. Therefore, mlr3 provides a convenience function to quickly generate design tables and instantiate resampling strategies in an exhaustive grid fashion: expand_grid().

# get some example tasks
tasks = mlr_tasks$mget(c("pima", "sonar", "spam"))

# get some measures: accuracy (acc) and area under the curve (auc)
measures = mlr_measures$mget(c("classif.acc", "classif.auc"))

# get a featureless learner and a classification tree
# let both learners predict probabilities
learners = mlr_learners$mget(c("classif.featureless", "classif.rpart"), predict_type = "prob")

# compare via 3-fold cross validation
resamplings = mlr_resamplings$mget("cv3")

# create a BenchmarkDesign object
design = expand_grid(tasks, learners, resamplings)
print(design)
##             task                     learner     resampling
## 1: <TaskClassif> <LearnerClassifFeatureless> <ResamplingCV>
## 2: <TaskClassif>       <LearnerClassifRpart> <ResamplingCV>
## 3: <TaskClassif> <LearnerClassifFeatureless> <ResamplingCV>
## 4: <TaskClassif>       <LearnerClassifRpart> <ResamplingCV>
## 5: <TaskClassif> <LearnerClassifFeatureless> <ResamplingCV>
## 6: <TaskClassif>       <LearnerClassifRpart> <ResamplingCV>

3.6.2 Execution and Aggregation of Results

After the benchmark design is ready, we can directly call benchmark()

# execute the benchmark
bmr = benchmark(design)

Note that we did not instantiate the resampling instance, but expand_grid() took care of it for us: each resampling strategy is instantiated for each task during the construction of the exhaustive grid.

After the benchmark, we can calculate and aggregate the performance with .$aggregate():

bmr$aggregate(measures)
##                hash  resample_result task_id          learner_id resampling_id
## 1: 7623a63355b86781 <ResampleResult>    pima classif.featureless           cv3
## 2: 63448c888a56793c <ResampleResult>    pima       classif.rpart           cv3
## 3: ec2cc9a91e50255e <ResampleResult>   sonar classif.featureless           cv3
## 4: bcc319fbeeaa365d <ResampleResult>   sonar       classif.rpart           cv3
## 5: ba29326bff0ab914 <ResampleResult>    spam classif.featureless           cv3
## 6: e69deb64b3be3e42 <ResampleResult>    spam       classif.rpart           cv3
##    classif.acc classif.auc
## 1:      0.6510      0.5000
## 2:      0.7331      0.7528
## 3:      0.5335      0.5000
## 4:      0.7114      0.7568
## 5:      0.6060      0.5000
## 6:      0.8983      0.9013

We can aggregate the results further. For example, we might be interested which learner performed best over all tasks. Since we have data.table object here, we could do the following:

bmr$aggregate(measures)[, list(acc = mean(classif.acc), auc = mean(classif.auc)), by = "learner_id"]
##             learner_id    acc    auc
## 1: classif.featureless 0.5968 0.5000
## 2:       classif.rpart 0.7809 0.8037

Alternatively, we can also use the tidyverse approach:

library("magrittr")
requireNamespace("dplyr")
requireNamespace("tibble")

bmr$aggregate(measures) %>%
  tibble::as_tibble() %>%
  dplyr::group_by(learner_id) %>%
  dplyr::summarise(acc = mean(classif.acc), auc = mean(classif.auc))

## # A tibble: 2 x 3
##   learner_id            acc   auc
##   <chr>               <dbl> <dbl>
## 1 classif.featureless 0.597 0.5  
## 2 classif.rpart       0.781 0.804

Unsurprisingly, the classification tree outperformed the featureless learner.

3.6.3 Converting specific benchmark objects to resample objects

A BenchmarkResult object is essentially a collection of multiple ResampleResult objects. As these are stored in a column of the aggregated data.table(), we can easily extract them:

tab = bmr$aggregate(measures)
rr = tab[task_id == "spam" & learner_id == "classif.rpart"]$resample_result[[1]]
print(rr)
## <ResampleResult> of 3 iterations
## Task: spam
## Learner: classif.rpart

We can now investigate this resampling and even single resampling iterations using one of the approach shown in the previous section:

rr$aggregate(measure)
## classif.ce 
##     0.1017

# get the iteration with worst AUC
worst = rr$performance(measures) %>%
  tibble::as_tibble() %>%
  dplyr::slice(which.min(classif.auc)) %>%
  dplyr::select(classif.auc, iteration)

# get the corresponding learner
lrn = rr$learners[[worst$iteration]]
lrn
## <LearnerClassifRpart:classif.rpart>
## Model: -
## Parameters: xval=0
## Packages: rpart
## Predict Type: prob
## Feature types: logical, integer, numeric, character, factor, ordered
## Properties: importance, missings, multiclass, selected_features,
##   twoclass, weights