3 Model Optimization

Model Tuning

Machine learning algorithms have default values set for their hyperparameters. Irrespective, these hyperparameters need to be changed by the user to achieve optimal performance on the given dataset. A manual selection of hyperparameter values is not recommended as this approach rarely leads to an optimal performance. To substantiate the validity of the selected hyperparameters (= tuning), data-driven optimization is recommended. In order to tune a machine learning algorithm, one has to specify (1) the search space, (2) the optimization algorithm (aka tuning method) and (3) an evaluation method, i.e., a resampling strategy and (4) a performance measure.

In summary, the sub-chapter on tuning illustrates how to:

This sub-chapter also requires the package mlr3tuning, an extension package which supports hyperparameter tuning.

Feature Selection

The second part of this chapter explains feature selection also known as variable selection. Feature selection is the process of finding a subset of relevant features of the available data. The purpose can be manifold:

  • Enhance the interpretability of the model,
  • speed up model fitting or
  • improve the learner performance by reducing noise in the data.

In this book we focus mainly on the last aspect. Different approaches exist to identify the relevant features. In this sub-chapter on feature selection, three approaches are emphasized:

  • Filter algorithms select features independently of the learner according to a score.
  • Variable importance filters select features that are important according to a learner.
  • Wrapper methods iteratively select features to optimize a performance measure.

Note, that filters do not require a learner. Variable importance filters require a learner that can calculate feature importance values once it is trained. The obtained importance values can be used to subset the data, which then can be used to train any learner. Wrapper methods can be used with any learner but need to train the learner multiple times.

Nested Resampling

In order to get a good estimate of generalization performance and avoid data leakage, both an outer (performance) and an inner (tuning/feature selection) resampling process are necessary. Following features are discussed in this chapter:

This sub-section will provide instructions on how to implement nested resampling, accounting for both inner and outer resampling in mlr3.