Nested cross-validation fits many models, and the outer folds are
independent of one another. nested_tune_grid() and its
siblings run those folds on mirai
daemons when a pool is connected, and serially otherwise. Nothing in the
call changes between the two. This page starts a pool of two daemons,
runs the getting-started guide’s loop on it, shows the result identical
to the same run made serially, and then goes through what the parallel
path checks before it dispatches, what it sends to each daemon, what to
do while developing the package, and how a run stops.
?nested_tune_grid carries the reference version of each
of those under its parallel section. This page shows them running.
The design and the workflow
These are the guide’s, unchanged: five outer folds of
mtcars, five inner folds under each, a random forest with
two parameters marked for tuning, and a grid of six candidates.
vignette("nested-cv") walks through each of them.
set.seed(1)
folds <- nested_resamples(
mtcars,
outside = vfold_cv(v = 5),
inside = vfold_cv(v = 5)
)
rf <- rand_forest(mtry = tune(), min_n = tune(), trees = 500) |>
set_engine("ranger") |>
set_mode("regression")
wf <- workflow(mpg ~ ., rf)
grid <- expand.grid(mtry = c(2L, 5L, 8L), min_n = c(2L, 10L))Starting a pool
Parallelism is switched on by connecting daemons and off by
disconnecting them. There is no argument on the loop functions for it.
mirai::daemons() starts the pool, and
mirai::status() reports how many daemons are connected. Two
or more is the threshold, the same one tune uses.
Each daemon is a separate R process. It does not inherit the options, library paths, or environment variables this session set after it was launched, and it loads nestedtune from an installed library rather than from this session. Both points matter below.
The pre-flight
Before any fold is dispatched, the call asks every connected daemon
whether it can load nestedtune, whether its copy defines the same
internal functions this session’s copy does, and whether it can load
each package the workflow and the tuner need, here parsnip, workflows
and ranger. A daemon that fails any of those stops the call with a
message naming how many daemons are affected and what is missing, rather
than letting the run go ahead and every fold come back as an opaque
worker failure. A daemon that does not answer within the bound is
reported as a non-response, which is a different message, so a slow
daemon is never told to install a package it already has. The bound is
thirty seconds by default and is an R option,
nestedtune.preflight_timeout, in milliseconds.
That first round trip is also what makes each daemon load the tidymodels stack, so the first parallel call after starting a pool is the slow one. Later calls in the same session reuse what the daemons already loaded.
Running the loop
The call is the guide’s call. With a pool connected the outer folds are dispatched to the daemons, and each fold’s inner tuning runs serially on its daemon, because parallelism inside a fold on top of parallelism across folds would oversubscribe the cores.
set.seed(2)
par_res <- nested_tune_grid(wf, folds, grid = grid)
par_res
#>
#> ── Nested cross-validation results ────────────────────────────────────
#> Outer resamples: 5-fold cross-validation
#> # A tibble: 5 × 9
#> splits id .metrics .selected .inner_metrics .notes
#> <list> <chr> <list> <list> <list> <list>
#> 1 <split [25/7]> Fold1 <tibble> <tibble> <tibble [12 × 8]> <tibble>
#> 2 <split [25/7]> Fold2 <tibble> <tibble> <tibble [12 × 8]> <tibble>
#> 3 <split [26/6]> Fold3 <tibble> <tibble> <tibble [12 × 8]> <tibble>
#> 4 <split [26/6]> Fold4 <tibble> <tibble> <tibble [12 × 8]> <tibble>
#> 5 <split [26/6]> Fold5 <tibble> <tibble> <tibble [12 × 8]> <tibble>
#> # ℹ 3 more variables: .completed <lgl>, .tuning_seed <int>,
#> # .outer_fit_seed <int>
#> ℹ Use `summary()` for what the run means: which folds failed, what
#> each one selected, and the estimate across them.The result carries no mark of how it was computed. That is deliberate, and the serial run below is why.
The same run, serially
The seed is the same, so the serial run is the same procedure on the same resamples. Each outer fold draws its tuning seed from the session state at entry, by its position in the design, before anything is dispatched, and each daemon then pins the same generator kind the serial path uses. So the fold’s inner tuning sees the same random stream whichever process runs it.
set.seed(2)
ser_res <- nested_tune_grid(wf, folds, grid = grid)
same <- c(
metrics = identical(par_res$.metrics, ser_res$.metrics),
selected = identical(par_res$.selected, ser_res$.selected),
tuning_seed = identical(par_res$.tuning_seed, ser_res$.tuning_seed)
)
same
#> metrics selected tuning_seed
#> TRUE TRUE TRUEidentical() on the outer scores, the selections and the
tuning seeds returns TRUE for each. The two results are identical, and
which one was computed on daemons is not recoverable from either. That
is what lets a pool be a matter of the session rather than of the
analysis: a script gives the same answer on a laptop with no daemons and
on a workstation with many.
What crosses the wire
A fold’s payload is its outer split, its inner resamples and its two seeds; the workflow, the grid and the tuner’s settings travel beside it. Every split in a design indexes one shared copy of the data, but serializing does not preserve that sharing, so sending the splits as they are would send one copy of the data per split. The data is therefore stripped from each split before dispatch and refilled on the daemon, and what crosses is the fold’s row indices plus one copy of the data per fold. Two things this does not reach, both of them objects you supply rather than anything the package builds. A recipe keeps a copy of the data it was created with, and a formula carries the environment it was written in, so a workflow built inside a function that holds a large object sends that object with every fold. Building the workflow at the top level avoids the second.
Developing against daemons
Daemons load nestedtune from an installed library. Under
devtools::load_all() the host session runs the source tree,
and the daemons cannot see it. With no installed copy in the library,
the pre-flight stops the call. When the library does hold one, the
daemons run that copy, which may be older than the source the host is
running. The pre-flight catches the case where the older copy is missing
a function the host’s copy has, and asks you to reinstall and restart
the pool. A running daemon keeps the namespace it already loaded, so
reinstalling underneath it changes nothing until the pool is
restarted.
For development, prime the daemons with the source tree instead:
mirai::everywhere(pkgload::load_all("path/to/nestedtune"))Interrupting and cancelling
Pressing interrupt at the console while folds are outstanding unwinds the blocking wait, restores the random-number state, and asks the daemons to stop the folds they have not finished, so the pool goes idle rather than computing results nobody will read. Stopping is a request. A fold already inside a compiled fitting routine may not be interruptible, and one that has nearly finished may simply finish. Nothing about an interrupted run is recorded, because a fold that was never given a chance to run has not failed.
Cancelling needs mirai’s dispatcher, the process that hands tasks to
daemons and can recall them. mirai::daemons(n) starts one
by default. A pool started with dispatcher = FALSE has no
dispatcher, so the request cannot reach the daemons and the outstanding
folds run to completion. The results of such a pool are correct, and it
is not refused. What it lacks is the ability to stop, and the call says
so at dispatch with a warning of class
nestedtune_pool_not_cancellable, once per run.
Here is that pool, with the warning caught by a handler and its class printed. The results it returns are the same as before.
mirai::daemons(2, dispatcher = FALSE)
caught <- NULL
set.seed(2)
nd_res <- withCallingHandlers(
nested_tune_grid(wf, folds, grid = grid),
nestedtune_pool_not_cancellable = function(w) {
caught <<- w
invokeRestart("muffleWarning")
}
)
class(caught)
#> [1] "nestedtune_pool_not_cancellable" "rlang_warning"
#> [3] "warning" "condition"
identical(nd_res$.metrics, ser_res$.metrics)
#> [1] TRUE
mirai::daemons(0)The message itself names the remedy: restart the pool with
mirai::daemons(n), which starts a dispatcher by
default.
These mirai daemons were started without a dispatcher, so this run
cannot be cancelled.
ℹ Interrupting it returns control to you, but the outer folds keep
computing on the pool and their results are never read.
ℹ For a pool that stops when you do, restart it with
`mirai::daemons(n)`, which starts a dispatcher by default.One case the call cannot tell apart is documented rather than guessed
at. Calling mirai::daemons(0) while folds are outstanding
produces exactly what a daemon dying mid-fold produces, so tearing the
pool down that way is recorded as fold failures rather than as a
cancellation. The other folds keep their results.
What this example cannot show
mtcars has 32 rows and the grid has 6 points, so each
fold takes a fraction of a second, and two daemons loading the
tidymodels stack cost more than the folds save. This page shows the
parallel path working and its result matching the serial one, not a
speedup. The speedup arrives where the folds are expensive: larger data,
a bigger grid, a slower engine, or preprocessing the loop has to redo in
every fold. The arithmetic in vignette("nested-cv") says
where the cost lives, and what is parallel here is the outer loop over
that cost.