4.1 Basic concept

4.1.1 Creating Task and Learner Objects

The first step is to generate the following mlr3 objects from the task dictionary and the learner dictionary, respectively:

  1. The classification task

    task = mlr_tasks$get("iris")
  2. A learner for the classification tree

    learner = mlr_learners$get("classif.rpart")

4.1.2 Setting up the train/test splits of the data (#split-data)

It is common to train on a majority of the data. Here we use 80% of all available observations and predict on the remaining 20% observations. For this purpose, we create two index vectors:

train_set = sample(task$nrow, 0.8 * task$nrow)
test_set = setdiff(seq_len(task$nrow), train_set)

4.1.3 Defining the “Experiment”

The process of fitting a machine-learning model, predicting on test data and scoring the predictions by comparing predicted and true labels is what we call an “experiment” in mlr. We start by initializing a new Experiment object by passing the created TaskClassif and LearnerClassif:

e = Experiment$new(task = task, learner = learner)
print(e)
## <Experiment> [defined]:
##  + Task: iris
##  + Learner: classif.rpart
##  - Model: [missing]
##  - Predictions: [missing]
##  - Performance: [missing]

The output shows a summary of the state of the experiment. We can see that the current state is “defined”. This means that the task and the learner have been stored, but nothing else happened so far.
If we query the state slot specifically, the (ordered) factor levels of the output show us all other possible states of an experiment:

e$state
## [1] defined
## Levels: undefined < defined < trained < predicted < scored

4.1.4 Training

Now we can finally train the learner on the task by calling the .$train() function of the experiment e we created:

e$train(row_ids = train_set)
## <Experiment> [trained]:
##  + Task: iris
##  + Learner: classif.rpart
##  + Model: [rpart]
##  - Predictions: [missing]
##  - Performance: [missing]
print(e)
## <Experiment> [trained]:
##  + Task: iris
##  + Learner: classif.rpart
##  + Model: [rpart]
##  - Predictions: [missing]
##  - Performance: [missing]

The output indicates that the Experiment object was modified (its state is now [trained]). Additionally, it was extended by a slot model:

print(e$model)
## n= 120 
## 
## node), split, n, loss, yval, (yprob)
##       * denotes terminal node
## 
## 1) root 120 78 virginica (0.3333 0.3167 0.3500)  
##   2) Petal.Length< 2.45 40  0 setosa (1.0000 0.0000 0.0000) *
##   3) Petal.Length>=2.45 80 38 virginica (0.0000 0.4750 0.5250)  
##     6) Petal.Width< 1.75 40  3 versicolor (0.0000 0.9250 0.0750) *
##     7) Petal.Width>=1.75 40  1 virginica (0.0000 0.0250 0.9750) *

4.1.5 Predicting

After the model was trained, we use the remaining part of the data for prediction. Remember that we initially split the data in train_set and test_set.

(Rather than using a subset of the initial data we could also pass a completetly new data.frame here.)

e$predict(row_ids = test_set)
## <Experiment> [predicted]:
##  + Task: iris
##  + Learner: classif.rpart
##  + Model: [rpart]
##  + Predictions: [PredictionClassif]
##  - Performance: [missing]
print(e)
## <Experiment> [predicted]:
##  + Task: iris
##  + Learner: classif.rpart
##  + Model: [rpart]
##  + Predictions: [PredictionClassif]
##  - Performance: [missing]

Now we gained a new slot named Predictions that we holds the prediction results. Since we are using R6 objects in mlr3, the result is again directly stored in object e.

A preview of the generated predictions can be retrieved by querying the prediction slot.

e$prediction
## <PredictionClassif> for 30 observations:
##     row_id     truth   response
##  1:      6    setosa     setosa
##  2:      7    setosa     setosa
##  3:      8    setosa     setosa
## ---                            
## 28:    131 virginica  virginica
## 29:    134 virginica versicolor
## 30:    148 virginica  virginica

If desired you can also extract the results to a normal data.table object:

e_dt = as.data.table(e$prediction)
head(e_dt)
##    row_id  truth response
## 1:      6 setosa   setosa
## 2:      7 setosa   setosa
## 3:      8 setosa   setosa
## 4:      9 setosa   setosa
## 5:     10 setosa   setosa
## 6:     13 setosa   setosa

4.1.5.1 Performance assessment

The last step of the experiment is quantifying the performance of the model by comparing the predicted labels with the true labels using a performance measure. The default measure for the classification tasks is the “mean misclassification error”.

task$measures
## [[1]]
## <MeasureClassifCE:classif.ce>
## Packages: Metrics
## Range: [0, 1]
## Minimize: TRUE
## Predict type: response

To conduct the performance assessment, we call the function .$score(). We now get a new slot called "performance" in our object:

e$score()
## <Experiment> [scored]:
##  + Task: iris
##  + Learner: classif.rpart
##  + Model: [rpart]
##  + Predictions: [PredictionClassif]
##  + Performance: classif.ce=0.06667

If we now print the Experiment object e again, we see that it consists of all six slots which store all important information of the experiment:

print(e)
## <Experiment> [scored]:
##  + Task: iris
##  + Learner: classif.rpart
##  + Model: [rpart]
##  + Predictions: [PredictionClassif]
##  + Performance: classif.ce=0.06667

4.1.5.2 Chaining methods

The underlying R6 class structure makes it possible to chain all of the operations above ($train(), $predict() and $score()) in a single call:

Experiment$new(task = task, learner = learner)$train(train_set)$predict(test_set)$score()
## <Experiment> [scored]:
##  + Task: iris
##  + Learner: classif.rpart
##  + Model: [rpart]
##  + Predictions: [PredictionClassif]
##  + Performance: classif.ce=0.06667