3.5 Feature Selection / Filtering
Often, data sets include a large number of features. The technique of extracting a subset of relevant features is called “feature selection.”
The objective of feature selection is to fit the sparse dependent of a model on a subset of available data features in the most suitable manner. Feature selection can enhance the interpretability of the model, speed up the learning process and improve the learner performance. Different approaches exist to identify the relevant features. Two different approaches are emphasized in the literature: one is called Filtering and the other approach is often referred to as feature subset selection or wrapper methods.
What are the differences (Guyon and Elisseeff 2003; Chandrashekar and Sahin 2014)?
Filtering: An external algorithm computes a rank of the features (e.g. based on the correlation to the response). Then, features are subsetted by a certain criteria, e.g. an absolute number or a percentage of the number of variables. The selected features will then be used to fit a model (with optional hyperparameters selected by tuning). This calculation is usually cheaper than “feature subset selection” in terms of computation time. All filters are connected via package mlr3filters.
Wrapper Methods: Here, no ranking of features is done. Instead, an optimization algorithm selects a subset of the features, evaluates the set by calculating the resampled predictive performance, and then proposes a new set of features (or terminates). A simple example is the sequential forward selection. This method is usually computationally very intensive as a lot of models are fitted. Also, strictly speaking, all these models would need to be tuned before the performance is estimated. This would require an additional nested level in a CV setting. After undertaken all of these steps, the final set of selected features is again fitted (with optional hyperparameters selected by tuning). Wrapper methods are implemented in the mlr3fselect package.
Embedded Methods: Many learners internally select a subset of the features which they find helpful for prediction. These subsets can usually be queried, as the following example demonstrates:
task = tsk("iris") learner = lrn("classif.rpart") # ensure that the learner selects features stopifnot("selected_features" %in% learner$properties) # fit a simple classification tree learner = learner$train(task) # extract all features used in the classification tree: learner$selected_features()## [1] "Petal.Length" "Petal.Width"
There are also Ensemble filters built upon the idea of stacking single filter methods. These are not yet implemented.
3.5.1 Filters
Filter methods assign an importance value to each feature. Based on these values the features can be ranked. Thereafter, we are able to select a feature subset. There is a list of all implemented filter methods in the Appendix.
3.5.2 Calculating filter values
Currently, only classification and regression tasks are supported.
The first step it to create a new R object using the class of the desired filter method.
Each object of class Filter has a .$calculate() method which computes the filter values and ranks them in a descending order.
library("mlr3filters")
filter = FilterJMIM$new()
task = tsk("iris")
filter$calculate(task)
as.data.table(filter)## feature score
## 1: Petal.Width 1.0000
## 2: Sepal.Length 0.6667
## 3: Petal.Length 0.3333
## 4: Sepal.Width 0.0000Some filters support changing specific hyperparameters.
This is similar to setting hyperparameters of a Learner using .$param_set$values:
filter_cor = FilterCorrelation$new()
filter_cor$param_set## <ParamSet>
## id class lower upper
## 1: use ParamFct NA NA
## 2: method ParamFct NA NA
## levels
## 1: everything,all.obs,complete.obs,na.or.complete,pairwise.complete.obs
## 2: pearson,kendall,spearman
## default value
## 1: everything
## 2: pearson# change parameter 'method'
filter_cor$param_set$values = list(method = "spearman")
filter_cor$param_set## <ParamSet>
## id class lower upper
## 1: use ParamFct NA NA
## 2: method ParamFct NA NA
## levels
## 1: everything,all.obs,complete.obs,na.or.complete,pairwise.complete.obs
## 2: pearson,kendall,spearman
## default value
## 1: everything
## 2: pearson spearmanRather than taking the “long” R6 way to create a filter, there is also a built-in shorthand notation for filter creation:
filter = flt("cmim")
filter## <FilterCMIM:cmim>
## Task Types: classif, regr
## Task Properties: -
## Packages: praznik
## Feature types: integer, numeric, factor, ordered3.5.3 Variable Importance Filters
All Learner with the property “importance” come with integrated feature selection methods.
You can find a list of all learners with this property in the Appendix.
For some learners the desired filter method needs to be set during learner creation.
For example, learner classif.ranger (in the package mlr3learners) comes with multiple integrated methods.
See the help page of ranger::ranger.
To use method “impurity,” you need to set the filter method during construction.
library("mlr3learners")
lrn = lrn("classif.ranger", importance = "impurity")Now you can use the mlr3filters::FilterImportance class for algorithm-embedded methods to filter a Task.
library("mlr3filters")
task = tsk("iris")
filter = flt("importance", learner = lrn)
filter$calculate(task)
head(as.data.table(filter), 3)## feature score
## 1: Petal.Length 46.831
## 2: Petal.Width 40.576
## 3: Sepal.Length 9.4623.5.4 Wrapper Methods
Wrapper feature selection is supported via the mlr3fselect extension package. At the heart of mlr3fselect are the R6 classes:
FSelectInstanceSingleCrit,FSelectInstanceMultiCrit: These two classes describe the feature selection problem and store the results.FSelector: This class is the base class for implementations of feature selection algorithms.
3.5.5 The FSelectInstance Classes
The following sub-section examines the feature selection on the Pima data set which is used to predict whether or not a patient has diabetes.
task = tsk("pima")
print(task)## <TaskClassif:pima> (768 x 9)
## * Target: diabetes
## * Properties: twoclass
## * Features (8):
## - dbl (8): age, glucose, insulin, mass, pedigree, pregnant, pressure,
## tricepsWe use the classification tree from rpart.
learner = lrn("classif.rpart")Next, we need to specify how to evaluate the performance of the feature subsets.
For this, we need to choose a resampling strategy and a performance measure.
hout = rsmp("holdout")
measure = msr("classif.ce")Finally, one has to choose the available budget for the feature selection.
This is done by selecting one of the available Terminators:
- Terminate after a given time (
TerminatorClockTime) - Terminate after a given amount of iterations (
TerminatorEvals) - Terminate after a specific performance is reached (
TerminatorPerfReached) - Terminate when feature selection does not improve (
TerminatorStagnation) - A combination of the above in an ALL or ANY fashion (
TerminatorCombo)
For this short introduction, we specify a budget of 20 evaluations and then put everything together into a FSelectInstanceSingleCrit:
library("mlr3fselect")
evals20 = trm("evals", n_evals = 20)
instance = FSelectInstanceSingleCrit$new(
task = task,
learner = learner,
resampling = hout,
measure = measure,
terminator = evals20
)
instance## <FSelectInstanceSingleCrit>
## * State: Not optimized
## * Objective: <ObjectiveFSelect:classif.rpart_on_pima>
## * Search Space:
## <ParamSet>
## id class lower upper levels default value
## 1: age ParamLgl NA NA TRUE,FALSE <NoDefault[3]>
## 2: glucose ParamLgl NA NA TRUE,FALSE <NoDefault[3]>
## 3: insulin ParamLgl NA NA TRUE,FALSE <NoDefault[3]>
## 4: mass ParamLgl NA NA TRUE,FALSE <NoDefault[3]>
## 5: pedigree ParamLgl NA NA TRUE,FALSE <NoDefault[3]>
## 6: pregnant ParamLgl NA NA TRUE,FALSE <NoDefault[3]>
## 7: pressure ParamLgl NA NA TRUE,FALSE <NoDefault[3]>
## 8: triceps ParamLgl NA NA TRUE,FALSE <NoDefault[3]>
## * Terminator: <TerminatorEvals>
## * Terminated: FALSE
## * Archive:
## <ArchiveFSelect>
## Null data.table (0 rows and 0 cols)To start the feature selection, we still need to select an algorithm which are defined via the FSelector class
3.5.6 The FSelector Class
The following algorithms are currently implemented in mlr3fselect:
- Random Search (
FSelectorRandomSearch) - Exhaustive Search (
FSelectorExhaustiveSearch) - Sequential Search (
FSelectorSequential) - Recursive Feature Elimination (
FSelectorRFE) - Design Points (
FSelectorDesignPoints)
In this example, we will use a simple random search.
fselector = fs("random_search")3.5.7 Triggering the Tuning
To start the feature selection, we simply pass the FSelectInstanceSingleCrit to the $optimize() method of the initialized FSelector. The algorithm proceeds as follows
- The
FSelectorproposes at least one feature subset and may propose multiple subsets to improve parallelization, which can be controlled via the settingbatch_size). - For each feature subset, the given
Learneris fitted on theTaskusing the providedResampling. All evaluations are stored in the archive of theFSelectInstanceSingleCrit. - The
Terminatoris queried if the budget is exhausted. If the budget is not exhausted, restart with 1) until it is. - Determine the feature subset with the best observed performance.
- Store the best feature subset as the result in the instance object.
The best feature subset (
$result_feature_set) and the corresponding measured performance ($result_y) can be accessed from the instance.
fselector$optimize(instance)## INFO [00:16:51.274] [bbotk] Starting to optimize 8 parameter(s) with '<FSelectorRandomSearch>' and '<TerminatorEvals> [n_evals=20]'
## INFO [00:16:51.461] [bbotk] Evaluating 10 configuration(s)
## INFO [00:16:53.772] [bbotk] Result of batch 1:
## INFO [00:16:53.775] [bbotk] age glucose insulin mass pedigree pregnant pressure triceps classif.ce
## INFO [00:16:53.775] [bbotk] TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE 0.2891
## INFO [00:16:53.775] [bbotk] FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE 0.2344
## INFO [00:16:53.775] [bbotk] FALSE TRUE FALSE TRUE FALSE FALSE FALSE TRUE 0.2539
## INFO [00:16:53.775] [bbotk] FALSE TRUE FALSE FALSE FALSE TRUE TRUE FALSE 0.2695
## INFO [00:16:53.775] [bbotk] FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE 0.2266
## INFO [00:16:53.775] [bbotk] FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE 0.2695
## INFO [00:16:53.775] [bbotk] FALSE FALSE FALSE TRUE TRUE FALSE TRUE FALSE 0.3008
## INFO [00:16:53.775] [bbotk] TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE 0.3867
## INFO [00:16:53.775] [bbotk] TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE 0.2852
## INFO [00:16:53.775] [bbotk] FALSE TRUE FALSE FALSE TRUE FALSE FALSE TRUE 0.2656
## INFO [00:16:53.775] [bbotk] uhash
## INFO [00:16:53.775] [bbotk] 0397bf4d-ed68-4189-b87d-665d3ba8bfb1
## INFO [00:16:53.775] [bbotk] bf43579f-5292-4bcf-80ca-d541c61073f4
## INFO [00:16:53.775] [bbotk] b218e67a-e2f8-47a6-aac4-051976f87539
## INFO [00:16:53.775] [bbotk] 6809a2d0-e4bb-485d-9dfd-48dc2dcb842c
## INFO [00:16:53.775] [bbotk] 5524d947-5513-4177-a56a-81e1203b05f9
## INFO [00:16:53.775] [bbotk] 6969a2b0-76f5-4c64-8062-299b2f7d4778
## INFO [00:16:53.775] [bbotk] b2e82c2a-6b3f-431f-a966-6f76d7ed349f
## INFO [00:16:53.775] [bbotk] ce737021-1805-46d0-a16b-b7b0bdebc141
## INFO [00:16:53.775] [bbotk] 5bf0fb43-ff39-4ddc-994c-b0af7e028ff2
## INFO [00:16:53.775] [bbotk] 1b3d65d0-fb54-4e8e-8536-0e747cb7305d
## INFO [00:16:53.776] [bbotk] Evaluating 10 configuration(s)
## INFO [00:16:55.928] [bbotk] Result of batch 2:
## INFO [00:16:55.930] [bbotk] age glucose insulin mass pedigree pregnant pressure triceps classif.ce
## INFO [00:16:55.930] [bbotk] TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE 0.2422
## INFO [00:16:55.930] [bbotk] TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE 0.2773
## INFO [00:16:55.930] [bbotk] TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE 0.2500
## INFO [00:16:55.930] [bbotk] TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE 0.2891
## INFO [00:16:55.930] [bbotk] TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE 0.3516
## INFO [00:16:55.930] [bbotk] TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE 0.2891
## INFO [00:16:55.930] [bbotk] TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE 0.2891
## INFO [00:16:55.930] [bbotk] FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE 0.3320
## INFO [00:16:55.930] [bbotk] TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE 0.3164
## INFO [00:16:55.930] [bbotk] TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE 0.2695
## INFO [00:16:55.930] [bbotk] uhash
## INFO [00:16:55.930] [bbotk] 819f956a-0291-40fc-9649-5ca2797cdb75
## INFO [00:16:55.930] [bbotk] bf409079-e1c9-420a-9193-7ba47b718272
## INFO [00:16:55.930] [bbotk] 32fdb7ef-7246-4f24-b6f8-22dfd57c6db2
## INFO [00:16:55.930] [bbotk] 850ac165-b328-498f-9553-22b2a3fbb29b
## INFO [00:16:55.930] [bbotk] 2beadd7f-7431-4ce5-a7ff-56b4d089eb0e
## INFO [00:16:55.930] [bbotk] 3106d129-d918-4ea2-b419-7882b7d5d845
## INFO [00:16:55.930] [bbotk] 3f1da9a5-ad58-472c-8cca-842669f98ae8
## INFO [00:16:55.930] [bbotk] cced37ff-7101-4222-a48f-b5dc32abdf2a
## INFO [00:16:55.930] [bbotk] b7a4faec-1d54-47ec-b886-e1ee81dd3b74
## INFO [00:16:55.930] [bbotk] e909af72-0f2b-4239-a33e-8c4422ea61ae
## INFO [00:16:55.935] [bbotk] Finished optimizing after 20 evaluation(s)
## INFO [00:16:55.936] [bbotk] Result:
## INFO [00:16:55.937] [bbotk] age glucose insulin mass pedigree pregnant pressure triceps
## INFO [00:16:55.937] [bbotk] FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE
## INFO [00:16:55.937] [bbotk] features x_domain classif.ce
## INFO [00:16:55.937] [bbotk] glucose,insulin,mass,pedigree,pressure <list[8]> 0.2266## age glucose insulin mass pedigree pregnant pressure triceps
## 1: FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE
## features x_domain classif.ce
## 1: glucose,insulin,mass,pedigree,pressure <list[8]> 0.2266instance$result_feature_set## [1] "glucose" "insulin" "mass" "pedigree" "pressure"instance$result_y## classif.ce
## 0.2266One can investigate all resamplings which were undertaken, as they are stored in the archive of the FSelectInstanceSingleCrit and can be accessed by using as.data.table():
as.data.table(instance$archive)## age glucose insulin mass pedigree pregnant pressure triceps classif.ce
## 1: TRUE TRUE TRUE TRUE TRUE TRUE FALSE TRUE 0.2891
## 2: FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE 0.2344
## 3: FALSE TRUE FALSE TRUE FALSE FALSE FALSE TRUE 0.2539
## 4: FALSE TRUE FALSE FALSE FALSE TRUE TRUE FALSE 0.2695
## 5: FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE 0.2266
## 6: FALSE FALSE FALSE TRUE TRUE TRUE TRUE TRUE 0.2695
## 7: FALSE FALSE FALSE TRUE TRUE FALSE TRUE FALSE 0.3008
## 8: TRUE FALSE FALSE FALSE FALSE FALSE FALSE TRUE 0.3867
## 9: TRUE TRUE TRUE FALSE TRUE TRUE TRUE TRUE 0.2852
## 10: FALSE TRUE FALSE FALSE TRUE FALSE FALSE TRUE 0.2656
## 11: TRUE TRUE TRUE TRUE FALSE TRUE TRUE TRUE 0.2422
## 12: TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE 0.2773
## 13: TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE 0.2500
## 14: TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE 0.2891
## 15: TRUE FALSE FALSE FALSE FALSE TRUE FALSE FALSE 0.3516
## 16: TRUE TRUE TRUE TRUE TRUE TRUE TRUE TRUE 0.2891
## 17: TRUE TRUE FALSE FALSE TRUE TRUE TRUE TRUE 0.2891
## 18: FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE 0.3320
## 19: TRUE FALSE TRUE TRUE TRUE TRUE FALSE TRUE 0.3164
## 20: TRUE TRUE TRUE TRUE TRUE FALSE TRUE TRUE 0.2695
## uhash timestamp batch_nr
## 1: 0397bf4d-ed68-4189-b87d-665d3ba8bfb1 2021-03-04 00:16:53 1
## 2: bf43579f-5292-4bcf-80ca-d541c61073f4 2021-03-04 00:16:53 1
## 3: b218e67a-e2f8-47a6-aac4-051976f87539 2021-03-04 00:16:53 1
## 4: 6809a2d0-e4bb-485d-9dfd-48dc2dcb842c 2021-03-04 00:16:53 1
## 5: 5524d947-5513-4177-a56a-81e1203b05f9 2021-03-04 00:16:53 1
## 6: 6969a2b0-76f5-4c64-8062-299b2f7d4778 2021-03-04 00:16:53 1
## 7: b2e82c2a-6b3f-431f-a966-6f76d7ed349f 2021-03-04 00:16:53 1
## 8: ce737021-1805-46d0-a16b-b7b0bdebc141 2021-03-04 00:16:53 1
## 9: 5bf0fb43-ff39-4ddc-994c-b0af7e028ff2 2021-03-04 00:16:53 1
## 10: 1b3d65d0-fb54-4e8e-8536-0e747cb7305d 2021-03-04 00:16:53 1
## 11: 819f956a-0291-40fc-9649-5ca2797cdb75 2021-03-04 00:16:55 2
## 12: bf409079-e1c9-420a-9193-7ba47b718272 2021-03-04 00:16:55 2
## 13: 32fdb7ef-7246-4f24-b6f8-22dfd57c6db2 2021-03-04 00:16:55 2
## 14: 850ac165-b328-498f-9553-22b2a3fbb29b 2021-03-04 00:16:55 2
## 15: 2beadd7f-7431-4ce5-a7ff-56b4d089eb0e 2021-03-04 00:16:55 2
## 16: 3106d129-d918-4ea2-b419-7882b7d5d845 2021-03-04 00:16:55 2
## 17: 3f1da9a5-ad58-472c-8cca-842669f98ae8 2021-03-04 00:16:55 2
## 18: cced37ff-7101-4222-a48f-b5dc32abdf2a 2021-03-04 00:16:55 2
## 19: b7a4faec-1d54-47ec-b886-e1ee81dd3b74 2021-03-04 00:16:55 2
## 20: e909af72-0f2b-4239-a33e-8c4422ea61ae 2021-03-04 00:16:55 2
## x_domain_age x_domain_glucose x_domain_insulin x_domain_mass
## 1: TRUE TRUE TRUE TRUE
## 2: FALSE TRUE TRUE FALSE
## 3: FALSE TRUE FALSE TRUE
## 4: FALSE TRUE FALSE FALSE
## 5: FALSE TRUE TRUE TRUE
## 6: FALSE FALSE FALSE TRUE
## 7: FALSE FALSE FALSE TRUE
## 8: TRUE FALSE FALSE FALSE
## 9: TRUE TRUE TRUE FALSE
## 10: FALSE TRUE FALSE FALSE
## 11: TRUE TRUE TRUE TRUE
## 12: TRUE TRUE TRUE TRUE
## 13: TRUE TRUE FALSE FALSE
## 14: TRUE TRUE TRUE TRUE
## 15: TRUE FALSE FALSE FALSE
## 16: TRUE TRUE TRUE TRUE
## 17: TRUE TRUE FALSE FALSE
## 18: FALSE FALSE FALSE FALSE
## 19: TRUE FALSE TRUE TRUE
## 20: TRUE TRUE TRUE TRUE
## x_domain_pedigree x_domain_pregnant x_domain_pressure x_domain_triceps
## 1: TRUE TRUE FALSE TRUE
## 2: TRUE FALSE TRUE TRUE
## 3: FALSE FALSE FALSE TRUE
## 4: FALSE TRUE TRUE FALSE
## 5: TRUE FALSE TRUE FALSE
## 6: TRUE TRUE TRUE TRUE
## 7: TRUE FALSE TRUE FALSE
## 8: FALSE FALSE FALSE TRUE
## 9: TRUE TRUE TRUE TRUE
## 10: TRUE FALSE FALSE TRUE
## 11: FALSE TRUE TRUE TRUE
## 12: TRUE FALSE FALSE FALSE
## 13: FALSE FALSE FALSE FALSE
## 14: TRUE TRUE TRUE TRUE
## 15: FALSE TRUE FALSE FALSE
## 16: TRUE TRUE TRUE TRUE
## 17: TRUE TRUE TRUE TRUE
## 18: FALSE FALSE TRUE FALSE
## 19: TRUE TRUE FALSE TRUE
## 20: TRUE FALSE TRUE TRUEThe associated resampling iterations can be accessed in the BenchmarkResult:
instance$archive$benchmark_result$data## <ResultData>
## Public:
## as_data_table: function (view = NULL, reassemble_learners = TRUE, convert_predictions = TRUE,
## clone: function (deep = FALSE)
## combine: function (rdata)
## data: list
## initialize: function (data = NULL, store_backends = TRUE)
## iterations: function (view = NULL)
## learners: function (view = NULL, states = TRUE, reassemble = TRUE)
## logs: function (view = NULL, condition)
## prediction: function (view = NULL, predict_sets = "test")
## predictions: function (view = NULL, predict_sets = "test")
## resamplings: function (view = NULL)
## sweep: function ()
## task_type: active binding
## tasks: function (view = NULL)
## uhashes: function (view = NULL)
## Private:
## deep_clone: function (name, value)
## get_view_index: function (view)The uhash column links the resampling iterations to the evaluated feature subsets stored in instance$archive$data(). This allows e.g. to score the included ResampleResults on a different measure.
Now the optimized feature subset can be used to subset the task and fit the model on all observations.
task$select(instance$result_feature_set)
learner$train(task)The trained model can now be used to make a prediction on external data.
Note that predicting on observations present in the task, should be avoided.
The model has seen these observations already during feature selection and therefore results would be statistically biased.
Hence, the resulting performance measure would be over-optimistic.
Instead, to get statistically unbiased performance estimates for the current task, nested resampling is required.
3.5.8 Automating the Feature Selection
The AutoFSelector wraps a learner and augments it with an automatic feature selection for a given task.
Because the AutoFSelector itself inherits from the Learner base class, it can be used like any other learner.
Analogously to the previous subsection, a new classification tree learner is created.
This classification tree learner automatically starts a feature selection on the given task using an inner resampling (holdout).
We create a terminator which allows 10 evaluations, and uses a simple random search as feature selection algorithm:
library("paradox")
library("mlr3fselect")
learner = lrn("classif.rpart")
terminator = trm("evals", n_evals = 10)
fselector = fs("random_search")
at = AutoFSelector$new(
learner = learner,
resampling = rsmp("holdout"),
measure = msr("classif.ce"),
terminator = terminator,
fselector = fselector
)
at## <AutoFSelector:classif.rpart.fselector>
## * Model: -
## * Parameters: xval=0
## * Packages: rpart
## * Predict Type: response
## * Feature types: logical, integer, numeric, factor, ordered
## * Properties: importance, missings, multiclass, selected_features,
## twoclass, weightsWe can now use the learner like any other learner, calling the $train() and $predict() method.
This time however, we pass it to benchmark() to compare the optimized feature subset to the complete feature set.
This way, the AutoFSelector will do its resampling for feature selection on the training set of the respective split of the outer resampling.
The learner then undertakes predictions using the test set of the outer resampling.
This yields unbiased performance measures, as the observations in the test set have not been used during feature selection or fitting of the respective learner.
This is called nested resampling.
To compare the optimized feature subset with the complete feature set, we can use benchmark():
grid = benchmark_grid(
task = tsk("pima"),
learner = list(at, lrn("classif.rpart")),
resampling = rsmp("cv", folds = 3)
)
# avoid console output from mlrfselect
logger = lgr::get_logger("bbotk")
logger$set_threshold("warn")
bmr = benchmark(grid, store_models = TRUE)
bmr$aggregate(msrs(c("classif.ce", "time_train")))## nr resample_result task_id learner_id resampling_id iters
## 1: 1 <ResampleResult[21]> pima classif.rpart.fselector cv 3
## 2: 2 <ResampleResult[21]> pima classif.rpart cv 3
## classif.ce time_train
## 1: 0.2865 0
## 2: 0.2565 0Note that we do not expect any significant differences since we only evaluated a small fraction of the possible feature subsets.