2.4 Train and Predict

In this section, we explain how tasks and learners can be used to train a model and predict to a new dataset. The concept is demonstrated on a supervised classification using the iris dataset and the rpart learner, which builds a singe classification tree.

Training a learner means fitting a model to a given data set. Subsequently, we want to predict the label for new observations. These predictions are compared to the ground truth values in order to assess the predictive performance of the model.

2.4.1 Creating Task and Learner Objects

The first step is to generate the following mlr3 objects from the task dictionary and the learner dictionary, respectively:

task = tsk("sonar")
1. A learner for the classification tree:
learner = lrn("classif.rpart")

2.4.2 Setting up the train/test splits of the data

It is common to train on a majority of the data. Here we use 80% of all available observations and predict on the remaining 20%. For this purpose, we create two index vectors:

train_set = sample(task$nrow, 0.8 * task$nrow)
test_set = setdiff(seq_len(task$nrow), train_set) 2.4.3 Training the learner The field $model stores the model that is produced in the training step. Before the $train() method is called on a learner object, this field is NULL: learner$model
## NULL

Next, the classification tree is trained using the train set of the iris task by calling the $train() method of the Learner: learner$train(task, row_ids = train_set)

This operation modifies the learner in-place. We can now access the stored model via the field $model: print(learner$model)
## n= 166
##
## node), split, n, loss, yval, (yprob)
##       * denotes terminal node
##
##  1) root 166 83 M (0.5000 0.5000)
##    2) V11>=0.1707 106 31 M (0.7075 0.2925)
##      4) V16< 0.6665 81 15 M (0.8148 0.1852)
##        8) V9>=0.09925 72  8 M (0.8889 0.1111) *
##        9) V9< 0.09925 9  2 R (0.2222 0.7778) *
##      5) V16>=0.6665 25  9 R (0.3600 0.6400)
##       10) V27< 0.5049 8  1 M (0.8750 0.1250) *
##       11) V27>=0.5049 17  2 R (0.1176 0.8824) *
##    3) V11< 0.1707 60  8 R (0.1333 0.8667) *

2.4.4 Predicting

After the model has been trained, we use the remaining part of the data for prediction. Remember that we initially split the data in train_set and test_set.

prediction = learner$predict(task, row_ids = test_set) print(prediction) ## <PredictionClassif> for 42 observations: ## row_id truth response ## 5 R M ## 7 R M ## 19 R R ## --- ## 198 M M ## 202 M M ## 204 M M The $predict() method of the Learner returns a Prediction object. More precisely, a LearnerClassif returns a PredictionClassif object.

A prediction objects holds the row ids of the test data, the respective true label of the target column and the respective predictions. The simplest way to extract this information is by converting the Prediction object to a data.table():

head(as.data.table(prediction))
##    row_id truth response
## 1:      5     R        M
## 2:      7     R        M
## 3:     19     R        R
## 4:     30     R        M
## 5:     34     R        R
## 6:     35     R        M

For classification, you can also extract the confusion matrix:

prediction$confusion ## truth ## response M R ## M 19 5 ## R 9 9 2.4.5 Changing the Predict Type Classification learners default to predicting the class label. However, many classifiers additionally also tell you how sure they are about the predicted label by providing posterior probabilities. To switch to predicting these probabilities, the predict_type field of a LearnerClassif must be changed from "response" to "prob" before training: learner$predict_type = "prob"

# re-fit the model
learner$train(task, row_ids = train_set) # rebuild prediction object prediction = learner$predict(task, row_ids = test_set)

The prediction object now contains probabilities for all class labels:

# data.table conversion
##    row_id truth response prob.M prob.R
## 1:      5     R        M 0.8889 0.1111
## 2:      7     R        M 0.8889 0.1111
## 3:     19     R        R 0.1333 0.8667
## 4:     30     R        M 0.8889 0.1111
## 5:     34     R        R 0.1333 0.8667
## 6:     35     R        M 0.8889 0.1111

# directly access the predicted labels:
head(prediction$response) ## [1] M M R M R M ## Levels: M R # directly access the matrix of probabilities: head(prediction$prob)
##           M      R
## [1,] 0.8889 0.1111
## [2,] 0.8889 0.1111
## [3,] 0.1333 0.8667
## [4,] 0.8889 0.1111
## [5,] 0.1333 0.8667
## [6,] 0.8889 0.1111

Analogously to predicting probabilities, many regression learners support the extraction of standard error estimates by setting the predict type to "se".

2.4.6 Plotting Predictions

Analogously to plotting tasks, mlr3viz provides a autoplot() method for Prediction objects. All available types are listed on the manual page of autoplot.PredictionClassif() or autoplot.PredictionClassif(), respectively.

library(mlr3viz)

learner = lrn("classif.rpart", predict_type = "prob")
learner$train(task) prediction = learner$predict(task)
autoplot(prediction)

autoplot(prediction, type = "roc")

library(mlr3viz)
library(mlr3learners)
local({ # we do this locally to not overwrite the objects from previous chunks
learner = lrn("regr.lm")
learner$train(task) prediction = learner$predict(task)
autoplot(prediction)
})

2.4.7 Performance assessment

The last step of modeling is usually the performance assessment. To assess the quality of the predictions, the predicted labels are compared with the true labels. How this comparison is calculated is defined by a measure, which is given by a Measure object. Note that if the prediction was made on a dataset without the target column, i.e. without true labels, then no performance can be calculated.

Predefined available measures are stored in mlr_measures (with convenience getter msr()):

mlr_measures
## <DictionaryMeasure> with 51 stored values
## Keys: classif.acc, classif.auc, classif.bacc, classif.ce,
##   classif.costs, classif.dor, classif.fbeta, classif.fdr, classif.fn,
##   classif.fnr, classif.fomr, classif.fp, classif.fpr, classif.logloss,
##   classif.mcc, classif.npv, classif.ppv, classif.precision,
##   classif.recall, classif.sensitivity, classif.specificity, classif.tn,
##   classif.tnr, classif.tp, classif.tpr, debug, oob_error, regr.bias,
##   regr.ktau, regr.mae, regr.mape, regr.maxae, regr.medae, regr.medse,
##   regr.mse, regr.msle, regr.pbias, regr.rae, regr.rmse, regr.rmsle,
##   regr.rrse, regr.rse, regr.rsq, regr.sae, regr.smape, regr.srho,
##   regr.sse, selected_features, time_both, time_predict, time_train

We choose accuracy (classif.acc) as a specific performance measure and call the method $score() of the Prediction object to quantify the predictive performance. measure = msr("classif.acc") prediction$score(measure)
## classif.acc
##       0.875

Note that, if no measure is specified, classification defaults to classification error (classif.ce) and regression defaults to the mean squared error (regr.mse).