Changelog • nestedtune

nestedtune 0.0.0.9000

vignette("estimate"), vignette("tuners"), vignette("results") and the parallel article are shorter and follow the shape of the tidymodels.org articles. vignette("estimate") now says where a held-out test set fits and how to choose the number of folds. vignette("tuners") gives each search one short section. vignette("results") reads one run from the estimate to the plots, and the column tour, the failed-fold run and the dplyr rules left it.
The help pages for reading a nested run are shorter and follow the shape of tune’s help pages. They cover the collect, compute and augment methods, agreement(), autoplot(), selection_rule(), the extractors, and the print, summary and predict methods. The condition classes and the lists of errors left these pages. ?collect_metrics.nested_results now states the standard error in three sentences and points to vignette("estimate").
Seven more help pages are shorter and follow the shape of tune’s help pages: nested_tune_bayes(), the two racing functions, nested_tune_sim_anneal(), nested_fit_resamples(), nested_workflow_map(), nested_final_fit() and nested_resamples(). The four loop pages point to ?nested_tune_grid for what the loop functions share. The condition classes left the pages, and most lists of errors did too, because each error message says what went wrong.
?nested_tune_grid, the nested cross-validation guide and the README opening are shorter and follow the shape of tune’s help pages and the tidymodels.org articles. The help page keeps what a user decides and what the result holds. The guide is one worked example. The shared sections on the other loop pages are shorter too.
The tuning results change for a design whose outer split repeats a row in its in_id. Each inner tuning call now gets its inner splits on the outer fold’s analysis set. It already did so for a design with no repeated row. Before, the inner splits stayed on the whole data. So tune finalized some parameter ranges on a frame that held the outer fold’s held-out rows. An example is a min_n() range finalized by row count. Each copy of a repeated row is its own row of the analysis set. Take an inner assessment set that the design leaves as the complement. It now holds every copy that the outer split holds of each row that the inner split does not train on. The next item, the fix for a leak of outer held-out rows, held each row once. That fix now covers a second case. There the outer split repeats no row. Its in_id holds an index that is fractional, negative or past the data. The fold still fails, and its inner metrics come from assessment sets that hold only rows the outer split holds.
Fixed a leak of outer held-out rows into inner tuning. Take an outer split whose in_id repeats a row, and an inner split on the outer split’s frame whose out_id is the logical NA. The inner assessment set was every row outside the inner in_id, so it held the outer fold’s held-out rows. Now it holds only rows that the outer split holds.
The nested tuning functions and nested_fit_resamples() now refuse a design whose outer or inner splits hold an NA in an in_id or out_id. The error has class nestedtune_bad_design, and it names each outer fold, inner split and slot that holds one. nested_resamples() refuses the same in a split of outside. Before, an NA in an outer in_id passed the entry check, and the fold failed inside rsample. An out_id that is the logical NA is still accepted, because rsample reads it as the complement.
The same refusal now covers three more shapes of row index. It refuses an in_id or out_id that is not a numeric vector, such as row numbers stored as character. It refuses a slot that holds a value outside integer range. It also refuses an out_id that is NULL or empty. nested_resamples() refuses the same in a split of outside. Before, a value such as 3e9 passed the entry check in most slots, and the fold failed inside rsample. The entry check refused it in an inner in_id of a nested_resamples() design. It named it as a row the outer split does not hold, with a coercion warning. A character index failed the fold in vctrs. An empty inner out_id was left out of tuning with no message. The out_id that is the logical NA is still accepted. The error names each split and slot, and says how the slot breaks the rule.
The README has a second table, which lists eight tuning functions from tune, finetune, workflowsets and tidyclust and the nestedtune function for each. Seven are supported, and tidyclust::tune_cluster() is not. The prose under the resampling table is shorter. The reason for each refused design now sits in the “Differences from rsample” section of ?nested_resamples. The error for an outer bootstrap no longer says that rsample::nested_cv() only warns about it, because rsample builds a group_bootstraps() call with no warning.
Four refusals of malformed designs changed, each with class nestedtune_bad_design.
- If an element of a splits column has the rsplit class but is not a list, it is now refused in either loop. check_nested() also refuses an element of the inner_resamples column that has the rset class but is not a list. Before, the check stopped with an R error such as “subscript out of bounds”.
- nested_resamples() refuses an element of outside, or of the rset that inside gives, that is not an rsplit at all. The refusal of an inner split that is a list but not an rsplit now says that it is a list.
- Take an inner split on the outer split’s frame whose two sets share only rows that the outer split holds out. It now gets the error that says it indexes rows its outer fold does not hold, which names the leak. Before, it got the error about shared rows.
- Take a split whose out_id is NA and whose class has a complement() method in the global environment. If every row that method returns is a row of the data, the rule on shared rows now reads those rows, as rsample does. Before, the rule read the rows of rsample’s own method.
The entry check runs faster on designs with many large inner splits.
A split whose assessment set holds a row of its analysis set is now refused in either loop, with class nestedtune_bad_design. Such a split scores the model on rows it trained on. Before, a split rebuilt with rsample::make_splits() escaped the refusals that read a design’s class. Such an outer split ran, and its fold was scored on its training rows. In the inner loop, an index into an outer analysis set counts as the data row it copies. An outer split can repeat a row, as a rebuilt bootstrap does. The design is then refused if an inner split puts one copy of that row in each set. An inner apparent split rebuilt with make_splits() under the id “Apparent” now gets this refusal. Before, the error told you to give it another id. The apparent split of an inner bootstrap design stays exempt under the id “Apparent”, because tune leaves it out of its estimates. nested_resamples(), the entry checks and the inner design that nested_final_fit() rebuilds all apply the rule. The rule also refuses rsample::rolling_origin() with lag above 0 in either loop, because each assessment set then holds the last lag analysis rows. Before, such a design ran. While the design keeps its lag setting, the error suggests lag = 0, with the lagged predictors built before resampling. A design cut to fewer rows loses that setting, and is refused with no hint.
An inner split whose id is missing is now refused, and so is an inner design in which two splits carry the same values in every id column. tune leaves the first split out of its estimates, and it miscounts the resamples of the second design. nested_final_fit() now applies these rules to the inner design it rebuilds on all the data. It also refuses loo_cv(), apparent() and permutations() designs there, where before it told an apparent() design to rename its split. Each refusal has class nestedtune_bad_design.
An inner split with the id “Apparent” is now refused unless it is the apparent split of a bootstrap design. tune leaves every split with that id out of its estimates, so before, such a fold tuned on fewer resamples than its design held, with nothing said. nested_tune_race_anova() and nested_tune_race_win_loss() also refuse the apparent split of a bootstrap, such as the one bootstraps(apparent = TRUE) adds. finetune eliminates race candidates on its score, which comes from the rows the model trained on. If a workflow in a set routes to a racer, nested_workflow_map() refuses such a design before any workflow runs. nested_final_fit() applies both rules to the inner design it rebuilds on all the data. Each refusal has class nestedtune_bad_design.
An apparent split beside bootstrap or permutation splits now needs the id “Apparent” to count as part of that design. tune leaves a split with that id out of its estimates. Under any other id, tune scores the split on the rows it trained on. So an inner bootstrap design whose apparent split carries another id is refused as an apparent() design. This holds whether the id was changed in place or the design was rebuilt with rsample::manual_rset(). nested_resamples() also refuses an inside that returns such a design, and names the outer fold. The error says that a bootstrap keeps its apparent split only under the id “Apparent”. Before, such an inner design ran and tune averaged that split’s score into the estimate. An outer design rebuilt with rsample::manual_rset() was refused before and still is, but its error now names that split as an apparent() row, not a bootstrap or permutation row.
A loo_cv(), apparent() or permutations() design, or an outer bootstrap, is now refused even when the design no longer carries that design’s class. nestedtune finds the design from the class that each split keeps. The seven functions that take a nested design refuse a row subset such as folds[1:2, ], a design rebuilt from its splits with rsample::manual_rset(), and a design with one such split among valid ones, in either loop. nested_resamples() refuses such a rebuilt or mixed outside, and refuses a row-subset outside as before, because it is not an rset. It also refuses an inside that returns such splits, and names the outer fold. Before, such an outer design and a rebuilt or mixed inner design ran unrefused. A row subset of an inner design got an error that called the column malformed. The new error has class nestedtune_bad_design and names the rsample function, as the outer bootstrap refusals now also do.
loo_cv(), apparent() and permutations() are now refused in either loop, because none gives a valid nested estimate there. nested_resamples() refuses them when it builds the design. The seven functions that take a nested design refuse them in a design built another way, such as with rsample::nested_cv(), before any fold runs. Before, most of these designs failed every outer fold only after the whole loop ran. The error has class nestedtune_bad_design and names the rsample function. The outer bootstrap refusal in nested_resamples() now has this class too. The README’s table marks the six designs as refused.
The README has a table of 15 rsample resampling functions. It says whether each one works as the outer loop and as the inner loop, and why the ones that do not work are refused or give no valid estimate. A design marked as working is one the tests run through nested_tune_grid().
collect_predictions() on a nested result or a workflow-set result gains summarize, which defaults to FALSE. With summarize = TRUE it returns one averaged prediction per data row, over every completed outer fold that held the row out. So on a repeated or Monte Carlo design, a row held out more than once gets one prediction there. The default per-fold table is unchanged. The average follows tune’s rules for summarize = TRUE. Because each fold selected its own candidate, the average spans those candidates, and the fold labels and .config are dropped. Quantile predictions are refused. The average refuses saved predictions whose .row column no longer holds exactly the rows its fold held out, each once. It also refuses three edited shapes that its rules cannot read: two or more factor outcome columns, a .pred_class column with no factor outcome, and a censored .pred entry that is not NULL and has no .eval_time. A NULL censored entry is left out of its row’s average. A metric computed on the averages describes several fitted models, not the tuning procedure, so collect_metrics() stays the nested estimate.
The per-fold tables of collect_metrics(summarize = FALSE) and compute_metrics(summarize = FALSE) carry the design’s fold labels each in its own column, as tune’s tables do. On a repeated design they now carry id and id2. Before, they carried one id that pasted the two together, such as "Repeat1, Fold1", and that pasted id is gone. A join on a repeated design’s folds uses both columns. The wide shape of collect_metrics() and the tables of a workflow-set result follow. The performance plot still labels its folds "Repeat1, Fold1".
summary() of a nested result, of a workflow-set result and of a final fit, and the final fit’s print, name the metric that chose the selected candidates. A line such as Selecting metric: rmse prints under the "best", "one_std_err" and "pct_loss" rules, the default included. It is absent under the "desirability" rule. The record that extract_procedure() returns holds the name as first_metric, and the summaries carry it as a component of the same name. The help for nested_workflow_map() says which metric of a metrics passed to it selects. The five tuning functions and nested_fit_resamples() now refuse a metric set that does not suit the model’s mode before any fold runs. An example is a classification metric on a regression model. nested_workflow_map() makes the same refusal for every workflow the map runs, before any workflow runs. Before, every fold ran and failed.
The help for metrics on nested_tune_grid() and its tuning siblings says which metric selects and which metrics the outer loop scores. The “Running the loop” section of the nested cross-validation guide says so too. Under the "best", "one_std_err" and "pct_loss" rules, the first metric in the set chooses each fold’s candidate. Every metric in the set is scored, so metric_set(mae, rmse) chooses by mae and also reports rmse. The help for metrics on nested_fit_resamples() says that every metric in the set is scored.
selection_rule() gains a fourth rule, "desirability", which selects with desirability2::select_best_desirability() over goals such as maximize(rsq) and minimize(num_comp). A goal can name a metric or a tuned parameter. nested_tune_grid(), nested_tune_bayes() and nested_final_fit() apply it. nested_tune_grid() and nested_tune_bayes() refuse at entry a goal that names neither a metric of the run nor a tuned parameter, with class nestedtune_selection_rule_unknown_term. The racing tuners and nested_tune_sim_anneal() refuse the rule with class nestedtune_selection_rule_unsupported. nested_tune_grid() and nested_tune_bayes() refuse it with the same class on a censored regression model. selection_rule() refuses a goal desirability2 refuses, with class nestedtune_selection_rule_terms. It refuses a goal that names a variable or a metric outside its first argument, with class nestedtune_selection_rule_term_arg. It refuses a limit under the rule, with class nestedtune_selection_rule_limit. The rule needs desirability2 0.2.0 or later, now in Suggests. Where it is absent, selection_rule(), nested_tune_grid(), nested_tune_bayes() and nested_final_fit() refuse the rule with class nestedtune_pkg_not_installed.
Every orchestrator and nested_final_fit() support an outer rsample::rolling_origin(), rsample::sliding_window(), rsample::sliding_index() or rsample::sliding_period() design with an inner rsample::rolling_origin() design. nested_tune_grid() and nested_fit_resamples() are tested on all four, and so is nested_final_fit() for their results. nested_tune_bayes(), nested_tune_race_anova(), nested_tune_race_win_loss() and nested_tune_sim_anneal() are tested on the rolling-origin design, and so is nested_final_fit() for their results, plus the sliding-window design for nested_tune_bayes() results. nested_workflow_map() is tested on the rolling-origin design. nested_resamples() builds all four designs with the same splits as rsample::nested_cv().
An outer rsample::rolling_origin() design with an inner rsample::sliding_window(), rsample::sliding_index() or rsample::sliding_period() design is tested under nested_tune_grid(), and so is nested_final_fit() for its results. nested_resamples() builds these three designs with the same splits as rsample::nested_cv(). These designs are also tested under nested_tune_bayes(), nested_tune_race_anova(), nested_tune_race_win_loss() and nested_tune_sim_anneal(), and so is nested_final_fit() for the results of those four tuners. nested_workflow_map() is tested on these three designs with its default fn. With fn set to each of those four tuners, it is tested on the sliding-period design alone. nested_fit_resamples() is tested to accept these designs. It checks the inner design but fits nothing on it. Any other inner design is not tested.
An outer rsample::sliding_window(), rsample::sliding_index() or rsample::sliding_period() design with an inner design of one of these three kinds gives nine pairs. nested_resamples() builds each pair with the same splits as rsample::nested_cv(), and each pair is tested under nested_tune_grid(). The pair of an outer and an inner rsample::sliding_window() is also tested under nested_tune_bayes(), nested_tune_race_anova(), nested_tune_race_win_loss(), nested_tune_sim_anneal() and nested_workflow_map() with its default fn. On that pair, nested_final_fit() is tested for nested_tune_grid() results, and nested_fit_resamples() is tested to accept the design. It checks the inner design but fits nothing on it. On the other eight pairs, no function but nested_resamples() and nested_tune_grid() is tested.
When an outer design leaves rows out of every assessment set, augment() now names the first five of those rows in its error, or all of them when there are five or fewer. The error names no design type. The condition class is still nestedtune_augment_rows.
nested_tune_grid(), nested_tune_bayes(), nested_tune_race_anova(), nested_tune_race_win_loss(), nested_tune_sim_anneal() and nested_fit_resamples() now take a parsnip model specification with a formula or a recipe, in tune’s order: nested_tune_grid(spec, preprocessor, resamples). The call gives the same result as the one on workflows::workflow(preprocessor, spec) under the same seed. A specification with no preprocessor, or with one that is not a formula or a recipe, is refused with class nestedtune_bad_preprocessor. A preprocessor given beside a workflow is refused with class nestedtune_preprocessor_with_workflow. nested_final_fit() still takes a workflow, and its message for a bare specification names workflows::workflow(preprocessor, spec).
A nested_final_fit now answers extract_fit_parsnip(), extract_fit_engine(), extract_recipe(), extract_mold(), extract_preprocessor(), extract_spec_parsnip() and outcome_names() with what the same call gives on its trained workflow. extract_recipe() takes estimated. Apart from extract_recipe(), the workflow methods drop an argument they do not know, so these refuse one instead.
A nested_final_fit now answers extract_parameter_set_dials() with the parameter set tune stored on its tuning run. The trained workflow holds no tune() placeholder, so the set is read from that run and not from the workflow. With a data-frame grid, the set can keep an unknown range such as that of mtry. A fit from nested_fit_resamples() is refused with class nestedtune_no_tuning_run.
collect_metrics() on a nested_results or a nested_results_set takes type = "wide", which returns one column per metric, as tune’s collect_metrics() does. A summarized table holds each metric’s mean, and an unsummarized one holds each outer fold’s estimate. The fold labels, .eval_time and wflow_id stay as key columns. An unknown type is refused with class nestedtune_bad_type. A metric scored with two estimators, or named like a key column, is refused with class nestedtune_wide_collision.
autoplot() on a nested_results or a nested_results_set takes metric and eval_time, which choose the panels of type = "performance". A static metric’s panel is drawn whatever eval_time names, as in tune. A metric or a time the run did not score, or either argument with type = "parameters", is refused with class nestedtune_bad_plot_filter.
predict() on a nested_results or a nested_results_set now refuses with class nestedtune_predict_results and names nested_final_fit(), which fits the model to predict with. Before, R gave its generic “no applicable method” error.
Five messages no longer break across two lines partway through a sentence. They used a line continuation that cli::format_inline() does not join, so the reader saw a newline and an indent. The refusal nested_final_fit() gives for reordered model arguments showed a stray backslash as well. The other four are three refusals of a malformed nested design and the completion line of a workflow set’s print.
The workflow identity a result records no longer depends on the number formatting the session happens to use. deparse() reads the scipen option, so a workflow holding a value such as 1e5 recorded one identity in a session that set the option and another in a session that did not, and nested_final_fit() refused a workflow the user had built from the same code. The option is now fixed while the identity is written, so a record made on one machine matches the workflow rebuilt on another.
The workflow identity every result records now holds a case-weights column, as written, and a tailor postprocessor, as its adjustments in order with each one’s type and arguments, when the workflow carries them. nested_final_fit() refuses a workflow that differs in either, with class nestedtune_workflow_mismatch, and the message names the part: the column, the adjustment and its argument, or one side carrying the part and the other not. A workflow with neither keeps the identity it had, so a results object saved before this change is accepted under the workflow it ran under. tailor joins Suggests for the tests.
Weights set on the design with tune::add_resample_weights() now reach the outer average. collect_metrics(), summary(), autoplot() and compute_metrics() report the weighted mean and tune’s weighted standard error over the folds that scored. The per-fold table from collect_metrics(summarize = FALSE) carries each fold’s weight in a .weight column. A fold that fails or scores NA is left out and the remaining weights are scaled to sum to one. A design without weights reports the numbers it did before.
compute_metrics() now refuses a completed fold whose saved predictions do not hold exactly the rows that fold held out, each once, before any fold is scored. The refusal has class nestedtune_compute_metrics_predictions and names each fold that fails. Five shapes this refuses are a missing row, a repeated .row, an NA .row, a row the fold did not hold out, and no .row column. A .row that is not a whole number is refused too. augment() refuses the same shapes. On a workflow-set run the message names the workflow.
The printed summary() of a run, the final fit’s print and its summary() now name a selection rule other than the default. The line reads Selected by: and then the rule’s name, orderings and limit, in the words the print of selection_rule() uses after its class tag. A workflow set’s printed summary adds the line inside each tuned workflow’s section. Both summaries gain a select component holding the rule.
The help for nested_tune_grid(), nested_tune_bayes(), nested_tune_race_anova(), nested_tune_race_win_loss(), nested_tune_sim_anneal() and nested_fit_resamples() now takes the text for grid, param_info, iter and metrics from tune’s and finetune’s own pages. What this package adds for one of those arguments reads in the page’s Details.
The note summary() prints for a run or a workflow set and the autoplot(type = "performance") subtitles now name the nested estimate as the number to report for the model nested_final_fit() builds. The help for nested_tune_bayes(), the racing tuners and nested_tune_sim_anneal() now says the same. The summary note limits this to a procedure chosen before seeing the estimate. For a fit built from a workflow set, the message a final fit prints names that workflow’s rows of collect_metrics() on the set as the number to report. This holds for a workflow chosen before seeing the set’s estimates.
compute_metrics() and augment() now answer on a run whose control set save_pred = TRUE, and on a workflow-set run. compute_metrics() scores each outer fold’s saved predictions with a metric set you give it. It summarizes them as collect_metrics() does, and it does not run the inner selection again. Its event_level defaults to the level the run recorded. augment() joins each data row’s out-of-fold prediction onto the data. On a design that holds each row out once, it joins the saved predictions as they are. A repeated v-fold or a Monte Carlo design can hold some row out more than once. There, every row joins the average that collect_predictions(summarize = TRUE) gives for it. On such a design it refuses saved quantile predictions, because that average does not cover them. It refuses there the three edited shapes that average refuses. It refuses an outer design that leaves some row out of every assessment set. It also refuses a fold whose saved predictions do not hold exactly the rows that fold held out, each once. A data column named like a prediction column is refused before any refusal of the average.
The coverage job no longer fails after every test passed. Daemons started by the test suite write their coverage traces to a directory of their own, and the job merges those traces afterwards, skipping any file a daemon was killed in the middle of writing.
Every help page now passes the same plain-English sweep as the guides: no semicolon, no contraction, no has been or have been, none of the words should, may, might, could and would, no word ending in -ing after a comma other than a few prepositions and pronouns, and none of the filler words and phrases, in any roxygen prose sentence.
The four guides, the parallel article and the README are rewritten to pass a plain-English sweep: no semicolon, no contraction, no has been or have been, none of the words should, may, might, could and would, no word ending in -ing after a comma other than a few prepositions and pronouns (including, during, nothing), and none of a list of filler words and phrases, in any prose sentence.
Every result records the model specification and the preprocessor of the workflow it ran under, as the workflow entry of the record extract_procedure() returns. The entry holds the model’s type, engine, mode and arguments, and the preprocessor, in deparsed form, with recipe step ids and the data left out. Case weights and a postprocessor are not part of the entry. nested_final_fit() compares the workflow it is handed against that entry and refuses one whose model or preprocessor differs, with class nestedtune_workflow_mismatch, naming the part that differs. A workflow rebuilt from the same code is accepted. A results object saved before this entry existed is refused as one from an earlier version.
Every help page is rewritten for someone who has run tune_grid() and never nested: no roxygen sentence over 30 words, no sentence naming more than four code spans, each page opening on what the reader gets from it, and the words “procedure”, “orchestrator” and “candidate” set up once on ?nested_tune_grid and either set up again or preceded by a link to that page wherever another page uses them. extract_scored_candidates(), agreement(), selection_rule() and extract_procedure() have new titles; nothing else about any page’s arguments or behaviour changed.
The four guides, the parallel article and the README are rewritten for someone who has run tune_grid() and never nested: no prose sentence over 30 words, the words “procedure” and “candidate” set up on each page before it relies on them, and the word “orchestrator” gone. Those pages, and this file, keep each explanation on one page with a link from the others. The site-only article “Why nest: a simulation” and its simulation script are removed; the estimate page’s cited measurements make the same point.
Every help page is rewritten for a tidymodels user: argument entries of at most two sentences, detail under section headings, the arguments an orchestrator shares with the tune, finetune or workflowsets function it wraps inherited from that page, and examples written with |> that run under R CMD check. ?nested_resamples carries the memory table the README used to.
The documentation site no longer carries pages built from files internal to the repository, and its build fails if any tracked markdown file other than the README, this file, the licence, the code of conduct or the contributing guide gets a top-level page.
autoplot() on a nested_results_set with type = "performance" adds a subtitle sentence counting the workflows whose average for a metric comes from fewer outer folds than the workflow completed, pointing at summary() for which metric and how many folds.
A nested_results_set keeps its class only while every row is one of the run’s own workflows, so dropping or reordering rows and adding columns keep it and a record column dropped or renamed, a repeated wflow_id or a row not the run’s own give a plain tibble. A warning or error raised for one workflow is the original condition with Workflow "<id>": in front of its message.
summary(), autoplot() and agreement() answer on a nested_results_set, each workflow’s view keyed by its wflow_id: summary() returns one summary per workflow and prints one section each, the performance plot puts the workflows along the x axis inside a panel per metric, the parameters plot draws a panel per workflow and tuned parameter, and agreement() stacks each workflow’s table under wflow_id.
nested_workflow_map() runs every workflow of a workflowsets::workflow_set() through one nested design in one call, with fn naming the orchestrator, the orchestrator’s arguments in ..., and the set’s option column overriding an argument for one workflow. It returns a nested_results_set, one nested_results per row, whose collect_*() readers stack each workflow’s table under wflow_id, and nested_final_fit(x, id = ) fits one workflow by its own record; workflowsets joins Suggests.
nested_fit_resamples() scores a workflow with nothing to tune on the outer folds of a nested design, the same outer loop with the inner stage removed, so a fixed workflow and a tuned one score on identical folds. Its result answers every reader, with .selected empty on every fold and extract_procedure() naming the tuner "fit_resamples".
The five tuning orchestrators refuse a workflow with no tune() marker at entry, with class nestedtune_untuned_workflow, naming nested_fit_resamples(), which in turn refuses a workflow carrying a marker with class nestedtune_tuned_workflow.
A select argument on the five tuning orchestrators names the rule each outer fold selects its candidate by, built by the new selection_rule(): "best" by default, or "one_std_err" and "pct_loss" with the parameter orderings tune’s rules take. The rule is recorded as extract_procedure(res)$select and nested_final_fit() selects by it.
save_pred and extract on a control passed as control reach the outer fit, adding a .predictions or .extracts list column to the result, and collect_predictions() and collect_extracts() stack them with the fold labels over the folds that completed. A run that did not ask for the column is refused with class nestedtune_column_not_saved.
extract_procedure() returns the record of what ran, the tuner, its arguments and the control as it took effect, from a results object and from a final fit alike.
The em dash is gone from every text the package publishes.
Three readers stack a per-fold list column into one table with the design’s fold labels beside it: collect_notes() over every outer fold, and collect_selections() and collect_inner_metrics() over the folds that completed, warning once with class nestedtune_partial_summary on a partial run.
tidymodels joins Suggests. Every vignette attaches it before nestedtune, behind a guard that ends the page with one notice when it or another of the page’s optional packages is absent.
The parallel article and three further vignettes are added. The parallel article runs the guide’s loop on two mirai daemons and shows the result identical to the serial run; vignette("results") reads the results object; vignette("tuners") runs the Bayesian, racing and annealing searches on the guide’s design; and vignette("estimate") says what the nested number estimates, leaving vignette("nested-cv") as the getting-started path.
The tuning orchestrators refuse a malformed resamples design at the call, before any fold runs, with class nestedtune_bad_design, naming every offending fold, split, index or column: an inner element that is not an rsplit or rset, an inner split over a frame other than the outer split’s own or its analysis set, an inner split indexing a row its outer fold holds out, an empty inner design, an NA or repeated label, and a label column that is not character or factor or not named id or id followed by a digit. Designs from nested_resamples() and rsample::nested_cv() pass unchanged.
Before any fold is dispatched to mirai daemons, the startup check asks every daemon for each package the workflow and the tuner need, and stops with class nestedtune_daemons_missing_pkgs naming how many daemons are affected and which packages. The host’s own entry check refuses a package the workflow needs that is not installed, a recipe step’s as well as the engine’s, under nestedtune_pkg_not_installed with an install.packages() call to paste.
extract_tune_results() and extract_scored_candidates() on an object they have no method for signal nestedtune_no_extract_method naming that object, whatever else was passed in ....
Printing the columnless type token vctrs::vec_cbind_frame_ptype() hands back writes the banner and the rows alone. A column add that leaves two columns sharing a record column’s name returns a bare tibble, and names<- keeps the object intact when each record column keeps its name and returns a bare tibble when one is renamed or shadowed.
On a nested_resamples() design, every inner tuning call sees only its outer fold’s analysis rows, so a param_info parameter whose range tune finalizes from the data is finalized without the rows that fold holds out, as it already was on an rsample::nested_cv() design. The candidates such a run searches, and so its selections and estimate, change from earlier development versions; nested_final_fit() still finalizes on the full data.
nested_final_fit() refuses a results object in which no outer fold completed, with class nestedtune_no_completed_folds, the class collect_metrics(), autoplot() and agreement() raise on the same object.
nested_tune_sim_anneal() runs the outer loop with finetune’s simulated annealing inside: each fold scores initial candidates on its inner resamples, then perturbs the current candidate for iter iterations, with a finetune::control_sim_anneal() passed as control. iter below 1 and initial below 1 or given as a tune_results are refused at entry, and control_sim_anneal(verbose_iter = FALSE) keeps a serial run quiet (#35).
nested_tune_race_anova() and nested_tune_race_win_loss() run the outer loop with finetune’s two racing tuners inside, taking nested_tune_grid()’s arguments and a finetune::control_race() as control; each fold’s .inner_metrics holds every candidate its race scored with n the inner resamples each was scored on. A package the race needs that is not installed, a control of the wrong kind, and an inner design with no more resamples than the control’s burn_in are refused at entry; finetune, lme4 and BradleyTerry2 join Suggests (#35).
Each outer fold of a nested_results carries its inner tuning run’s metrics as .inner_metrics, tune::collect_metrics() of that fold’s run with one row per candidate and metric, plus .iter from the iterating searches. A fold that scored nothing carries a zero-row table, and extract_scored_candidates() no longer carries .eval_time (#57).
nested_tune_grid() and nested_tune_bayes() take a control object passed as control, tune::control_grid() or tune::control_bayes(), and pass it to the inner tuning call in every fold, with allow_par forced to FALSE and the Bayesian seed set to each fold’s tuning seed. A control of the wrong kind is refused with class nestedtune_bad_control, and any other name or an unnamed value in ... with nestedtune_bad_dots (#33, #35).
nested_final_fit() takes the workflow and the nested_results object, nested_final_fit(object, results), and re-runs the procedure that result recorded: the design’s inner resampling specification, the tuner and its arguments, the metric set and the selection rule. A results object with no such record, one that is no longer a nested_results, or one with no rows is refused before any fitting with class nestedtune_bad_results.
predict() and augment() work directly on a nested_final_fit, returning what the same call on extract_workflow(final) returns; augment is re-exported. augment() takes new_data and eval_time only and refuses anything else.
print() and summary() on a nested_final_fit name the procedure that ran: the candidates scored for a grid search, and for an iterating search the initial candidates and iterations, each as what ran beside what was requested.
nested_tune_bayes() runs the outer loop with tune::tune_bayes() as the inner tuner, scoring initial space-filling candidates on each fold’s inner resamples and letting a Gaussian process propose up to iter more under objective. The Gaussian process is seeded from the fold’s tuning seed, so a fold reproduces from its .tuning_seed alone, and initial is a count only, since one tuning run cannot serve every outer fold.
Every nested_results carries a procedure attribute recording the tuner that ran, its own arguments, and param_info, event_level and eval_time, which travels with the class and is shed with it. The metric set travels beside it as attr(x, "metrics"), absent when none was passed.
agreement() reports how often each candidate was selected across the outer folds: one row per distinct combination of selected values, with n and prop over the completed folds, most frequent first. The most frequent combination is not the final model’s parameters.
print() on a nested_results accepts n and width and passes them to the rendering of the outer-fold rows, matched by full name only.
nested_tune_grid() takes an eval_time argument, passed to every tune call whose answer depends on it, so a censored regression workflow scored by a dynamic survival metric is measured at the times named. A metric measured at several times is summarized per time by collect_metrics() and summary() in an .eval_time column and drawn one panel per time by autoplot(type = "performance").
summary() on a nested_final_fit reports the full-data tuning run’s resampling scheme, the candidates it scored and the values selection chose. Where a number would be, it names the nested estimate as the number to report for this model.
Printing a nested_results shows the object: its outer folds as the tibble rows they are, the resampling scheme, a count of the folds that did not complete, and a note when the folds did not all search the same grid. Printing never warns and never errors, a run in which no fold completed included.
summary() on a nested_results returns an object holding the requested and completed fold counts, the failed folds with the stage each failed at, the values the completed folds selected, and the metric estimates over the folds that completed. A partial run warns and still returns the summary, and one where every fold failed does the same.
A nested_results whose record of its fold-label columns cannot label its rows names each fold by its row position.
An operation that changes which outer folds a nested_results holds returns a plain tibble: slice(), head(), x[1, ], a filter() that drops a fold, bind_rows(), and dropping any of the columns the run is recorded in. Reordering rows with arrange(), adding a column with mutate() or bind_cols(), reordering columns with relocate(), and a left_join() that matches one row apiece keep the class and the run’s record.
The same rule covers the vctrs verbs and base rbind(): vctrs::vec_slice(x, 1), vctrs::vec_rbind(), vctrs::vec_c() and rbind() return a plain tibble, while reordering rows with vec_slice() and adding a column with vctrs::vec_cbind(x, extra) keep the class. vec_cbind() and bind_cols() build on the first argument’s type, so with the results object second they return a plain tibble.
dplyr::rename() moving one of the columns the run is recorded in returns a plain tibble.
dplyr and vctrs are hard dependencies; both were already installed alongside nestedtune through tune.
A results object records the columns its resampling design labelled the folds with, so a column you add is read as a fold label only when the design itself carries a column of that name. Before, the fold-label columns were worked out from the column names, so adding id_extra reported the folds as Fold1, x.
Replacing a fold-label column with a value that cannot be ordered, as dplyr::mutate(x, id = list(c(1, 2), 3, 4)) does, returns a plain tibble rather than failing inside the class rule.
The packages a workflow declares are attached inside each mirai daemon before any fold is dispatched, so a recipe with unqualified selectors such as all_numeric_predictors() resolves on a worker as it does serially.
nested_tune_grid(), nested_final_fit() and nested_resamples() take ... immediately after their required arguments, so grid, metrics and param_info must be named and a mistyped or unsupported argument is an error naming the function it was passed to. Every method the package registers whose ... is documented as unused refuses an argument the same way.
collect_metrics() on a nested_results takes summarize after ..., matching tune’s own method, so it must be named.
nested_tune_grid() takes param_info, passed unchanged to tune::tune_grid() on every outer fold, and refuses one that is not a dials::parameters() object before the first fold is fitted.
nested_tune_grid() takes event_level, naming which level of a two-class outcome factor counts as the event, and sends it to the inner tuning run and the outer scoring fit alike. A value that is not "first" or "second" is refused before the first fold is fitted.
The documentation site builds with the tidymodels organization’s shared pkgdown theme, and the organization’s contributing guide and code of conduct build as pages of the site.
The package has moved to the tidymodels organization, at https://github.com/tidymodels/nestedtune with its documentation site at https://nestedtune.tidymodels.org/. The old repository address redirects; the old documentation address does not.
A parallel run refuses to start when a daemon is holding an older install of nestedtune, with class nestedtune_daemons_incompatible, naming the functions its copy is missing and the fix: reinstall, then restart the pool. A running daemon keeps the version it has already loaded, so reinstalling underneath one changes nothing.
A parallel run started on a pool that cannot be cancelled, one from mirai::daemons(n, dispatcher = FALSE), warns once at the start of the run with class nestedtune_pool_not_cancellable. Its results are correct; what the pool lacks is the ability to stop on an interrupt.
Running the outer folds in parallel sends each fold one copy of the data instead of one copy per inner resample, the splits being emptied before dispatch and refilled on the daemon. Results are unchanged, and the serial path is untouched.
The object nested_final_fit() returns has two accessors for the tuning run behind it: extract_tune_results() returns that run, and extract_scored_candidates() the candidate settings it scored. Every metric inside that run was computed on the resamples that chose the candidate it describes, so it is not the model’s performance.
nested_tune_grid() and nested_final_fit() refuse a workflow that has a model but no preprocessor, pointing at workflows::add_formula(), add_recipe() and add_variables(), and a workflow with no model, pointing at workflows::add_model().
When nested_final_fit() cannot re-run a design’s stored inner specification, the error names the caller’s call rather than an internal function.
nested_resamples() refuses an inside specification that does not produce an rset.
When a resampling specification fails to evaluate, the error names the specification that was tried instead of deparsing the data into the message, and is wrapped in a nestedtune error carrying the original as its cause.
The reference pages and the vignettes are built into a documentation website, rebuilt whenever a change lands on the default branch that the package itself can see.
Interrupting a parallel run asks the folds already sent to the daemons to stop, so the pool goes idle rather than computing results nobody will read. Cancelling needs mirai’s dispatcher, which mirai::daemons(n) starts by default, and a fold already inside a compiled fitting routine may not be interruptible.
The check that runs before parallel dispatch asks every connected daemon whether it can load the package, and a daemon that does not answer is reported as a non-response, class nestedtune_daemons_no_response, rather than as one that cannot load the package, class nestedtune_daemons_cannot_load. Both also carry nestedtune_daemons_unusable.
The wait for that check is settable with options(nestedtune.preflight_timeout = <milliseconds>), a single positive, finite number, 30 seconds by default. The first parallel call after starting a cold pool is the slow one, since it is what makes each daemon load the package.
Cancelling a parallel run stops it, raising a nestedtune_cancelled condition that inherits from nestedtune_interrupted and returning nothing, with the caller’s RNG state restored. Calling mirai::daemons(0) while folds are outstanding is recorded as fold failures instead, since it produces exactly what a daemon dying mid-fold produces.
autoplot() draws a nested cross-validation result. The default view puts one point per outer fold at the value that fold’s inner tuning selected, one panel per tuned parameter, and type = "performance" draws each outer fold’s score with a dashed line at the nested estimate, read from the same place collect_metrics() reads it.
Both views keep every outer fold that was attempted on the axis, a fold that failed leaving a visible gap. The subtitle says how much of the requested design ran, and each panel says when fewer folds contributed to it than completed, as mtry (2 of 3 chose) or rmse (from 2 folds).
ggplot2 is a hard dependency. Plotting a run where no outer fold completed, or asking for the parameters view of a design with no tuned parameters, is refused with a message saying which it was.
nested_tune_grid() runs its outer folds in parallel when mirai daemons are connected, with no argument to set, and inner tuning stays serial because nesting parallelism inside parallelism oversubscribes cores. Parallel results are identical to serial ones, because each fold’s seeds are drawn before the loop starts and assigned by position.
nested_final_fit() builds the model to deploy by running the same tuning procedure the nested estimate describes with the whole dataset in hand, and returns it as a separate object. The nested estimate is the number to report for that model. collect_metrics(), show_best() and select_best() refuse a final fit rather than returning something that reads as its score.
Because the inner resampling specification is stored unevaluated and re-evaluated at final-fit time, it is written with literal arguments, inside = vfold_cv(v = 5) rather than inside = vfold_cv(v = k). A specification whose variables have gone out of scope fails with a message naming it.
An outer fold that fails does not end the run: the remaining folds still run, .completed marks the failed one, and .notes says which stage failed and why, carrying tune’s own notes about the cause. A fold that completes on only part of its inner design keeps the notes explaining what was lost.
nested_tune_grid() checks a data-frame grid against the workflow before fitting anything, refusing by name a column that is not marked for tuning or a tuned parameter with no column.
collect_metrics() summarizes only the outer folds that completed, warns naming the ones that did not, and errors rather than returning NA when none completed.
nested_tune_grid() runs the nested cross-validation loop end to end: for each outer fold it tunes on that fold’s inner resamples with tune::tune_grid(), selects a candidate, finalizes the workflow, and fits and scores it on the outer split. The result keeps each fold’s chosen parameters alongside its metrics, so disagreement between folds is visible rather than averaged away, and collect_metrics() returns the per-fold metrics or their summary across outer folds.
nested_tune_grid() is reproducible from a single set.seed() before the call: it derives one tuning seed and one outer-fit seed per fold up front, so a fold’s result depends on its position in the design rather than on the order folds run in, and it leaves the caller’s random-number state as it found it.
nested_resamples() builds a nested resampling design without keeping a copy of the data for every outer fold, selecting the same rows as rsample::nested_cv() for the same seed and specifications. On a 20000-row dataset with a five-fold inner resampling, a 50-fold outer design holds 10× the source data rather than 57×.
nested_resamples() refuses an outer bootstrap, since the same observation can otherwise land in both the inner analysis and the inner assessment set.