The nested estimate measures how well a whole tuning procedure (tune,
select, fit) performs on new data. The worked example is in
vignette("nested-cv").
Why the tuned score is optimistic
When we tune with cross-validation, we score several candidates (the settings tried) and keep the best. Scores are noisy, so the best of several tends to look better than it is. The winner’s score overstates how the model will do on new data.
Nested resampling moves the tuning inside each outer fold. Each fold tunes, selects and fits on its own analysis rows, then scores the chosen setting once on assessment rows that the tuning never saw.
What the estimate describes
The mean from collect_metrics() is the error of the
procedure when it trains on each fold’s analysis rows and predicts new
data.
The estimate describes the procedure, not one fitted model, and we
report it for the deployed model because nested_final_fit()
runs that procedure on all the data. No rows remain that the final fit
did not see, so there is no second number to compute.
The estimate is not a score of the parameter values the final fit chose. It is also the error on this data set, not the average over every training set the same source can produce. Luo and Barber (2026) prove that any assumption-free test of that average needs far more data than the training set holds, so this package offers no such test.
Where a test set fits
A test set gives one number from one split. The outer folds give a mean and show how much the score moves between folds. We can keep both.
We split with initial_split() and run the nested loop
and nested_final_fit() on training(). Then we
score the final model’s predictions on testing() once, as
last_fit() does after ordinary tuning. We report the test
score for that fitted model, and the nested estimate for the procedure
that made it.
Choosing the number of folds
Arlot and Celisse (2010) report that 5 to 10 folds is the usual advice for model selection, which is the inner loop’s job. They add that the best count depends on the problem.
More outer folds give each model more training rows, and each outer
fold adds a whole tuning run. vignette("nested-cv") counts
the fits.
A slight pessimism
Each outer model trains on fewer rows than the deployed model, so the estimate tends to look a little worse than the procedure deserves. Varma and Simon (2006) ran nested cross-validation on 40 samples with no signal, where the true error was 50.0%. The nested estimate was 54.2%, and they attribute the gap to training on 39 rows rather than 40.
On 41,121 hospital visits, Wilimitis and Walsh (2023) found nested cross-validation among the most pessimistic methods, by 1 to 2% of AUROC and 5 to 9% of AUPR. The gap is a property of the method, so it is no reason to adjust the reported number.
The standard error
std_err is the standard error of the mean across the
outer folds. It measures the precision of the mean, not the spread of
the folds. It treats the fold scores as independent, but the folds share
training rows, so it tends to understate the uncertainty. Bengio and
Grandvalet (2004) prove that no estimator of this variance is unbiased
in every case, so the package builds no interval from it.
Comparing workflows
A gap between two nested estimates is a reason to look closer, never
a result. Bayle et al. (2026) show that two stable procedures can have
an unstable difference. Picking the best of a set from
nested_workflow_map() is itself a selection that no outer
loop scored, so the package ranks none.
When the folds disagree
When the folds choose different values for a parameter, the data does not pin that value down. The estimate still holds, because it averages over that variation. For two similar candidates, the comparison itself is unstable (Bayle et al., 2026), so which one wins a fold says little.
When nesting pays
Nesting matters most when the search had room to overfit. Tibshirani and Tibshirani (2009) tuned on pure noise, where the true error is 50%. With 400 rows and 100 features, the tuned score was under 3 points too good.
With 40 rows and 1000 features, nearest shrunken centroids read 38.4%, 11.6 points too good, while an SVM was 2.5 points too good and a tree 0.2. That paper’s remedy is a correction, not nesting.
Nesting adds little with many rows and a small grid. Wilimitis and Walsh (2023) found no gain there, only extra cost. More rows alone do not remove the bias. Vabalas et al. (2019) saw cross-validation without nesting beat chance on pure noise at 1000 rows.
Feature selection belongs in the recipe, so the loop repeats it in every fold. A step run on the data before the call is not repeated, and no result can show it. Vabalas et al. (2019) found that selection outside the loop does more harm than tuning outside it.
References
Arlot, S., & Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys, 4, 40–79.
Bayle, A., Janson, L., & Mackey, L. (2026). The relative instability of model comparison with cross-validation. arXiv:2508.04409.
Bengio, Y., & Grandvalet, Y. (2004). No unbiased estimator of the variance of K-fold cross-validation. Journal of Machine Learning Research, 5, 1089–1105.
Luo, Y., & Barber, R. F. (2026). The limits of assumption-free tests for algorithm performance. Bernoulli, 32(3), 2427–2450.
Tibshirani, R. J., & Tibshirani, R. (2009). A bias correction for the minimum error rate in cross-validation. The Annals of Applied Statistics, 3(2), 822–829.
Vabalas, A., Gowen, E., Poliakoff, E., & Casson, A. J. (2019). Machine learning algorithm validation with a limited sample size. PLoS ONE, 14(11), e0224365.
Varma, S., & Simon, R. (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 7, 91.
Wilimitis, D., & Walsh, C. G. (2023). Practical considerations and applied examples of cross-validation for model development and evaluation in health care: Tutorial. JMIR AI, 2, e49023.