nested_final_fit() runs the tuning procedure once more with the whole
dataset in hand: it re-evaluates the design's inner resampling specification
against every row, tunes with tune::tune_grid(), selects the best
candidate, finalizes the workflow, and fits it on all the data. The result is
the model to deploy.
Arguments
- object
A
workflows::workflow()with at least one parameter marked for tuning withtune::tune(). Ordinarily the same workflow passed tonested_tune_grid().- resamples
A nested resampling design, from
nested_resamples()orrsample::nested_cv(). Only its inner specification and its data are used — the outer folds play no part in a final fit — but the whole design is still checked, so a designnested_tune_grid()refuses is refused here too: itssplitscolumn must holdrsplitobjects and itsinner_resamplescolumn anrsetper outer fold. The reverse does not follow: this function additionally needs the design's stored inner specification, which the loop never re-runs, so a design with none is refused here and runs perfectly well there.- grid
A data frame of candidate parameter values, or a positive whole number giving the size of a grid to generate. Passed to
tune::tune_grid().- metrics
A
yardstick::metric_set(), orNULLto use tune's defaults for the model's mode. The first metric in the set selects the best candidate.
Value
An object of class nested_final_fit with elements workflow (the
trained workflow, better reached with extract_workflow()), selected
(the parameters chosen), tuning (the tuning run they were chosen from),
and tuning_seed and fit_seed (the two seeds that reproduce it).
Details
The procedure a nested estimate describes is "resample this dataset by the inner specification, tune, select, fit", and the dataset that procedure is meant to be applied to is all of yours. So the final model comes from running it again with nothing held out — the same convention as cross-validating a model and then refitting on everything, one level up.
The outer folds play no part. Their selections are not pooled or voted on: they belong to the estimate, which describes the procedure across the instability those selections reveal, and not to this model.
What to report
Report the estimate from collect_metrics() on the nested_tune_grid()
result as this model's performance. That number estimates the k-fold test
error of the whole tune-and-fit procedure that produced this model, measured
on data no part of the procedure ever touched. Expect it to run slightly
pessimistic: each outer fold trains on its analysis rows alone, so every
model it scores is built on less data than this one. Varma and Simon (2006)
measured a 4.2-point overshoot from that effect at n = 40, and Wilimitis and
Walsh (2023) about 1-2% of AUROC on 41,121 records. The offset shrinks with
fold size and is not a correction to apply.
The model in hand has no honest number of its own. Everything computable from
its training data was consumed by selection or by fitting, including the
resampling metrics inside the tuning run stored on this object: those are
selection-time quantities, optimistically biased as a performance claim, and
collect_metrics() on x$tuning will hand them over without saying so. They
are kept because they are the record of what selection saw, not because they
describe this model.
Two things the nested estimate does not say. It is marginal over selection, not conditional on the parameters this model happens to carry, so it is not a claim about this configuration specifically. And it describes new data drawn like your training data — not a different population, and not a model retrained at a different size.
If the outer folds disagreed about the best parameters, report that too.
Reproducibility
Seed the session before the call, as elsewhere in tidymodels; there is no
seed argument. On entry the function draws two seeds in a single
sample.int(.Machine$integer.max, 2) call. The first covers building the
inner resamples and tuning; the second covers the final fit. Both are
applied with the generator kind pinned, and both are returned on the object.
The run is reproducible by hand from those two seeds alone:
set.seed(fit$tuning_seed, kind = "Mersenne-Twister",
normal.kind = "Inversion", sample.kind = "Rejection")
inner <- <the design's `inside` specification>(data)
tuned <- tune_grid(object, inner, grid = grid, metrics = metrics,
control = control_grid(allow_par = FALSE))
final <- finalize_workflow(object, select_best(tuned, metric = <first metric>))
set.seed(fit$fit_seed, kind = "Mersenne-Twister",
normal.kind = "Inversion", sample.kind = "Rejection")
fit(final, data)Building the resamples sits inside the first seed's scope, not before it:
constructing an rset draws from the generator, so a version that built them
earlier would still be reproducible from the session seed but no longer from
the two seeds above.
The caller's RNG state and generator kind are restored on exit, including
when the call errors. One consequence worth knowing: two consecutive calls
with no set.seed() between them return identical results, exactly as
repeated tune::tune_grid() calls do.
This binds randomness that flows through R's generator. Engines that randomize outside it — kernlab's SVMs, the deep-learning engines — cannot be pinned by any R-side scheme, here or in tune.
The inner specification is re-evaluated
A nested design stores its inside argument as an unevaluated call, and this
function evaluates it again — against the whole dataset, in the environment
you call from, not the one the design was built in.
Write it with literal arguments. inside = vfold_cv(v = 5) is re-evaluated
identically anywhere. inside = vfold_cv(v = k) is not: if k is gone by
the time you call this, you get an error naming the specification, and if
some other k is in scope you silently get a different design. Building a
design inside a function that parameterizes its resampling is the common way
to meet this.
References
Varma, S., & Simon, R. (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 7, 91.
Wilimitis, D., & Walsh, C. G. (2023). Practical considerations and applied examples of cross-validation for model development and evaluation in health care: Tutorial. JMIR AI, 2, e49023.
Examples
data(mtcars)
rec <- recipes::step_pca(
recipes::recipe(mpg ~ ., data = mtcars),
recipes::all_predictors(),
num_comp = tune::tune()
)
wf <- workflows::workflow(rec, parsnip::linear_reg())
set.seed(1)
folds <- nested_resamples(
mtcars,
outside = rsample::vfold_cv(v = 3),
inside = rsample::vfold_cv(v = 3)
)
# The estimate: what the procedure achieves.
set.seed(2)
res <- nested_tune_grid(wf, folds, grid = data.frame(num_comp = 1:3))
collect_metrics(res)
#> # A tibble: 2 × 5
#> .metric .estimator mean n std_err
#> <chr> <chr> <dbl> <int> <dbl>
#> 1 rmse standard 3.23 3 0.316
#> 2 rsq standard 0.722 3 0.112
# The model: what you deploy. Report the estimate above for it.
set.seed(3)
final <- nested_final_fit(wf, folds, grid = data.frame(num_comp = 1:3))
final
#>
#> ── Nested cross-validation final fit ───────────────────────────────────────────
#> Selected: num_comp = 1
#>
#> ℹ This model has no performance estimate of its own. Report the nested estimate
#> from `collect_metrics()` on the `nested_tune_grid()` result, which describes
#> the procedure that produced it.
#> ℹ Compare the parameters above with `.selected` from that run. Outer folds
#> choosing differently is selection instability, and it is information about
#> the procedure rather than noise.
#> ℹ `extract_tune_results()` returns the tuning run selection came from, and
#> `extract_scored_candidates()` the candidates it scored. Any metric reachable
#> through the first is a selection-time quantity, optimistically biased as a
#> claim about this model.
predict(extract_workflow(final), new_data = mtcars[1:3, ])
#> # A tibble: 3 × 1
#> .pred
#> <dbl>
#> 1 23.1
#> 2 23.1
#> 3 25.2