Nested Cross-Validation for Tidymodels • nestedtune

The best score from ordinary tuning is optimistic, because the same resamples pick the winner. nestedtune runs nested cross-validation instead. Each outer fold tunes a workflow on its own inner resamples with tune or finetune, then fits the chosen settings and scores them on rows the tuning never saw. The mean of the outer scores estimates how well the whole procedure (tune, select, fit) performs on new data. Report it for the model that nested_final_fit() builds.

Installation

# install.packages("pak")
pak::pak("tidymodels/nestedtune")

Example

library(tidymodels)
library(nestedtune)

set.seed(1)
folds <- nested_resamples(
  mtcars,
  outside = vfold_cv(v = 5),
  inside = vfold_cv(v = 5)
)

wf <- workflow(
  mpg ~ .,
  rand_forest(mtry = tune(), min_n = tune()) |>
    set_engine("ranger") |>
    set_mode("regression")
)
grid <- expand.grid(mtry = c(2L, 5L, 8L), min_n = c(2L, 10L))

set.seed(2)
res <- nested_tune_grid(wf, folds, grid = grid)

# The number to report for the model you deploy.
collect_metrics(res)
#> # A tibble: 2 × 5
#>   .metric .estimator  mean     n std_err
#>   <chr>   <chr>      <dbl> <int>   <dbl>
#> 1 rmse    standard   2.46      5  0.445 
#> 2 rsq     standard   0.844     5  0.0267

# The model to deploy, the same procedure run once more on all the data.
set.seed(3)
final <- nested_final_fit(wf, res)
predict(final, new_data = mtcars[1:3, ])
#> # A tibble: 3 × 1
#>   .pred
#>   <dbl>
#> 1  20.9
#> 2  20.9
#> 3  23.8

Supported resampling designs

The table covers 15 resampling functions from rsample. It says whether each one works as the outer loop and as the inner loop of a nested design.

Function	Outer loop	Inner loop
`vfold_cv()`	Yes	Yes
`mc_cv()`	Yes	Yes
`group_vfold_cv()`	Yes	Yes
`group_mc_cv()`	Yes	Yes
`clustering_cv()`	Yes	Yes
`bootstraps()`	Refused	Yes
`group_bootstraps()`	Refused	Yes
`loo_cv()`	Refused	Refused
`apparent()`	Refused	Refused
`validation_set()`	Yes¹	No
`permutations()`	Refused	Refused
`rolling_origin()`	Yes	Yes
`sliding_window()`	Yes	Yes
`sliding_index()`	Yes	Yes
`sliding_period()`	Yes	Yes

Yes means that the test suite runs the design in that loop through nested_tune_grid(), with the arguments those tests use, and that every outer fold completes. The design in the other loop is a v-fold design, or a time-series design for the four time-series rows.

Refused means that nestedtune stops with an error before any fold runs, whether nested_resamples() or rsample::nested_cv() built the design. No means that the design cannot be built in that loop. The “Differences from rsample” section of ?nested_resamples gives the reason for each Refused and No cell, and the rules for a single split in either loop.

Supported tuning functions

The table lists tuning functions from tune, finetune, workflowsets and tidyclust, and the nestedtune function that does each one’s job on a nested design.

Function	nestedtune function	Supported
`tune::tune_grid()`	`nested_tune_grid()`	Yes
`tune::tune_bayes()`	`nested_tune_bayes()`	Yes
`tune::fit_resamples()`	`nested_fit_resamples()`	Yes
`finetune::tune_race_anova()`	`nested_tune_race_anova()`	Yes
`finetune::tune_race_win_loss()`	`nested_tune_race_win_loss()`	Yes
`finetune::tune_sim_anneal()`	`nested_tune_sim_anneal()`	Yes
`workflowsets::workflow_map()`	`nested_workflow_map()`	Yes
`tidyclust::tune_cluster()`		No

Yes means that nestedtune exports a nested counterpart that does the row’s job on a nested design. The test suite runs it with vfold_cv() in both loops, and every outer fold completes. nested_fit_resamples() and nested_workflow_map() run no search of their own. The first scores a workflow with nothing to tune, and the second runs each workflow of a set through one of the other six functions. No means that no nestedtune function does the row’s job. For tune_cluster(), nothing yet settles what a nested estimate of a clustering metric means.

The resampling table’s Yes cells were tested through nested_tune_grid(). The “Time-series designs” section of ?nested_resamples says which time-series designs the other six functions are tested on. Choosing the inner tuner runs the Bayesian search, the two racing searches and simulated annealing on one example.

Learn more:

Nested cross-validation, the path from a design to a write-up.
What the estimate means, which quantity the nested number is and how far to trust it.
Choosing the inner tuner, the Bayesian search, the two racing searches, simulated annealing and a set of workflows on one design.
Reading the results, every column of the results object and every function that reads it.
Running the outer loop in parallel, the same call on a pool of mirai daemons.