6.2 Adding new Measures
In this section we showcase how to implement a custom performance measure.
A good starting point is writing down the loss function independently of mlr3 (we also did this in the mlr3measures package). Here, we illustrate writing measure by implementing the root of the mean squared error for regression problems:
= function(truth, response) {
root_mse = mean((truth - response)^2)
mse sqrt(mse)
}
root_mse(c(0, 0.5, 1), c(0.5, 0.5, 0.5))
## [1] 0.4082
In the next step, we embed the root_mse()
function into a new R6 class inheriting from base classes MeasureRegr
/Measure
.
For classification measures, use MeasureClassif
.
We keep it simple here and only explain the most important parts of the Measure
class:
= R6::R6Class("MeasureRootMSE",
MeasureRootMSE inherit = mlr3::MeasureRegr,
public = list(
initialize = function() {
$initialize(
super# custom id for the measure
id = "root_mse",
# additional packages required to calculate this measure
packages = character(),
# properties, see below
properties = character(),
# required predict type of the learner
predict_type = "response",
# feasible range of values
range = c(0, Inf),
# minimize during tuning?
minimize = TRUE
)
}
),
private = list(
# custom scoring function operating on the prediction object
.score = function(prediction, ...) {
= function(truth, response) {
root_mse = mean((truth - response)^2)
mse sqrt(mse)
}
root_mse(prediction$truth, prediction$response)
}
) )
This class can be used as template for most performance measures. If something is missing, you might want to consider having a deeper dive into the following arguments:
properties
: If you tag you measure with the property"requires_task"
, theTask
is automatically passed to your.score()
function (don’t forget to add the argumenttask
in the signature). The same is possible with"requires_learner"
if you need to operate on theLearner
and"requires_train_set"
if you want to access the set of training indices in the score function.aggregator
: This function (defaulting tomean()
) controls how multiple performance scores, i.e. from different resampling iterations, are aggregated into a single numeric value ifaverage
is set to micro averaging. This is ignored for macro averaging.predict_sets
: Prediction sets (subset of("train", "test")
) to operate on. Defaults to the “test” set.
Finally, if you want to use your custom measure just like any other measure shipped with mlr3 and access it via the mlr_measures
dictionary, you can easily add it:
::mlr_measures$add("root_mse", MeasureRootMSE) mlr3
Typically it is a good idea to put the measure together with the call to mlr_measures$add()
in a new R file and just source it in your project.
## source("measure_root_mse.R")
msr("root_mse")
## <MeasureRootMSE:root_mse>
## * Packages: -
## * Range: [0, Inf]
## * Minimize: TRUE
## * Properties: -
## * Predict type: response