6.1 Parallelization

Parralelization refers to the proccess of running mulitple jobs in parallel, simultanously. This process allows for significant savings in computing power.

mlr3 uses the future backends for parallelization. Make sure you have installed the required packages future and future.apply:

mlr3 is capable of parallelizing a variety of different scenarios. One of the most used cases is to parallelize the Resampling iterations. See Section Resampling for a detailed introduction to resampling.

In the following section, we will use the spam task and a simple classification tree ("classif.rpart") to showcase parallelization. We use the future package to parallelize the resampling by selecting a backend via the function future::plan(). We use the "multiprocess" backend here which uses threads on UNIX based systems and a “Socket” cluster on Windows.

future::plan("multiprocess")

task = tsk("spam")
learner = lrn("classif.rpart")
resampling = rsmp("subsampling")

time = Sys.time()
resample(task, learner, resampling)
Sys.time() - time

By default all CPUs of your machine are used unless you specify argument workers in future::plan().

On most systems you should see a decrease in the reported elapsed time. On some systems (e.g. Windows), the overhead for parallelization is quite large though. Therefore, it is advised to only enable parallelization for resamplings where each iteration runs at least 10s.

Choosing the parallelization level

If you have are transitioning from mlr, you might be used to selecting different parallelization levels, e.g. for resampling, benchmarking or tuning.

In mlr3 this is no longer required All kind of events are rolled out on the same level - there is no need to decide whether you want to parallelize the tuning OR the resampling. Just lean back and let the machine do the work :-)