Repository navigation

Adding `pred_measure` feature - #363

Draft

florence-bockting wants to merge 185 commits into

loo-v3.0.0from

pred_measure

Commits on Jul 2, 2026
- feat: add predictive performance measures as pred_measure* family
  Florence Bockting
  committed
  1c5d871
- feat: update print method for new pred_measure* family
  Florence Bockting
  committed
  bad8062
- feat: deprecate loo functions in favor of new pred_measure family
  Florence Bockting
  committed
  6d404e1
- tests: update tests for deprecated functions
  Florence Bockting
  committed
  f5854c2
- docs: add migration guide to loo package
  Florence Bockting
  committed
  c42c803
- docs: update documentation for new pred_measure functions and deprecated functions
  Florence Bockting
  committed
  7750200
- chore: update DESCRIPTION, NAMESPACE, NEWS and _pkgdown
  Florence Bockting
  committed
  1926429
Commits on Jul 3, 2026
- tests: add test data
  Florence Bockting
  committed
  02ecbbb
- tests: add unit tests for pred_measure function family
  Florence Bockting
  committed
  a24fee1
- fix: update elpd and fix typo in .compute_base_measure
  Florence Bockting
  committed
  3a670cf
- fix:update measure names in .validate_control
  Florence Bockting
  committed
  439867e
- docs: add vignettes for new predictive preformance measure
  Florence Bockting
  committed
  883dda0
- docs: add/update cross-references to vignettes
  Florence Bockting
  committed
  755984d
- chore: add data fitting to CI and avoid committing large data files
  Florence Bockting
  committed
  b260a32
- chore: add cli to Imports and lme4 to Suggests
  Florence Bockting
  committed
  082d7be
- fix: minor fixes
  Florence Bockting
  committed
  e394c30
- chore: add developer-notes
  Florence Bockting
  committed
  e9f1a3a
- add print method of test_pred_measure
  Florence Bockting
  committed
  cf0c79b
- chore: update developer-notes and add old-vs.-new API comparison
  Florence Bockting
  committed
  5cb9f10
- fix: small fixes to pass test checks
  Florence Bockting
  committed
  562b141
- Merge branch 'loo-v3.0.0' into pred_measure
  florence-bockting
  authored
  023add0
Commits on Jul 4, 2026
- tests: update snapshots
  Florence Bockting
  committed
  8346344
Commits on Jul 6, 2026
- tests: add snapshots for pred_measure_builtin
  Florence Bockting
  committed
  e99aeac
- refactor: add class loo to loo_pred_measure
  Florence Bockting
  committed
  884df6e
Commits on Jul 7, 2026
- docs: update pkg-info
  Florence Bockting
  committed
  b924ac5
- refactor: update loo_compare to work with loo_pred_measure
  Florence Bockting
  committed
  89d5e9a
- refactor: update pred_measure API for integration with loo_compare()
  Florence Bockting
  committed
  7ce2d50
- tests: new tests for loo_compare with loo_pred_measure
  Florence Bockting
  committed
  7394ba3
- docs: update documentation and glossary
  Florence Bockting
  committed
  494d24f
- docs: update documentation
  Florence Bockting
  committed
  9611656
- chore: update developer-notes
  Florence Bockting
  committed
  a3019e3
Commits on Jul 8, 2026
- refactor: rename revert_sign arg to higher_is_better
  Florence Bockting
  committed
  f64cf6c
- docs: update renaming arg to higher_is_better
  Florence Bockting
  committed
  441f0ea
- tests: update renaming to higher_is_better
  Florence Bockting
  committed
  b0015c7
- chore: update renaming higher_is_better argument
  Florence Bockting
  committed
  076e2c7
- Merge branch 'pred_measure' into integrate-loo_compare
  florence-bockting
  authored
  3e1bf93
- refactor: clean-up refactoring and adjust for renamed arg
  Florence Bockting
  committed
  06d6281
- fix: update missing argument and corresponding docs
  Florence Bockting
  committed
  b46ff58
- chore: update NEWS.md
  Florence Bockting
  committed
  f1fcaf0
Commits on Jul 16, 2026
- Merge branch 'loo-v3.0.0' into pred_measure
  florence-bockting
  authored
  674746e
- Merge branch 'pred_measure' into integrate-loo_compare
  florence-bockting
  authored
  e17e46d
Commits on Aug 12, 2026
- refactor: update computation of SE for loo_compare with pred_measure_loo
  Florence Bockting
  committed
  abc74b5
Commits on Aug 25, 2026
- refactor: create model_compare and workout design
  Florence Bockting
  committed
  faa27c5
Commits on Aug 27, 2026
- update model_compare() functionality
  Florence Bockting
  committed
  3e078aa
Commits on Aug 28, 2026
- Merge branch 'loo-v3.0.0' into pred_measure
  florence-bockting
  authored
  a1e66e5
- Merge branch 'loo-v3.0.0' into pred_measure
  florence-bockting
  authored
  0d1e072
- Merge branch 'pred_measure' into integrate-loo_compare
  florence-bockting
  authored
  8897e1f
- refactor: split model_compare.R and drop duplicated loo_compare internals
  
  Florence Bockting
  and
  claude
  committed
  1eb66de
- fix: restore the simplify argument to print.compare.loo
  
  Florence Bockting
  and
  claude
  committed
  166dab6
- review: update the model_compare function
  Florence Bockting
  committed
  c0c1fc0
- update: deprecate loo_compare but maintain backwards comptability
  Florence Bockting
  committed
  c594818
- review: code refactoring review
  Florence Bockting
  committed
  aa30e54
Commits on Aug 29, 2026
- docs: update function documentation to roxygen2 8.0.0
  Florence Bockting
  committed
  b1f0b32
- docs: update glossary to match loo_compare and model_compare
  Florence Bockting
  committed
  63c16a8
- fix: throw deprecation warning for loo_compare only once per session.
  Florence Bockting
  committed
  d874044
- fix: remove old 'estimates_only' label
  Florence Bockting
  committed
  9e930e3
- refactor: avoid duplicate computation
  Florence Bockting
  committed
  4fe4b0d
- docs: add loo_compare and model_compare to seealso statement
  Florence Bockting
  committed
  88890b6
- refactor: add warning for disagreeing measure name in attribute and list
  Florence Bockting
  committed
  51b3c9c
- review: minor style corrections and doc adjustment
  Florence Bockting
  committed
  81ae8eb
- review: removed dead code
  Florence Bockting
  committed
  fed3433
- vignette: update tutorial on model comparison
  Florence Bockting
  committed
  585b77c
- chore: update warning messages
  Florence Bockting
  committed
  3f63357
- review: update function docs and inline small helper
  Florence Bockting
  committed
  5307f23
Commits on Aug 30, 2026
- fix: computation of weights for (s)rps
  Florence Bockting
  committed
  b0f94cb
- fix: refactor computation of ic and mlpd
  Florence Bockting
  committed
  7184cae
- fix: computation of weights for (s)rps
  Florence Bockting
  committed
  91f251a
- tests: add tests for mlpd, ic, and updated weigts for (s)rps
  Florence Bockting
  committed
  757021e
- fix: check equality of log_weights from loo and psis_object
  Florence Bockting
  committed
  f08afcc
- fix: pass name of measures correctly
  Florence Bockting
  committed
  806600b
- review: add guards for potential silent failures
  Florence Bockting
  committed
  0c85ba8
- fix: kfold diagnostics, dims on update, reuse stored log_weights
  Florence Bockting
  committed
  d0b52fa
- fix: bacc pointwise path, dims after 3-D conversion, duplicate filter
  Florence Bockting
  committed
  1b1d050
- fix: error on group_ids until grouped summaries exist
  Florence Bockting
  committed
  8c46b00
- refactor: extract .lppd_from_inputs() from elpd, mlpd and ic
  Florence Bockting
  committed
  e74df04
- refactor: extract .acc_pointwise() from acc and bacc
  Florence Bockting
  committed
  b3a7db7
- refactor: extract .point_error_from_inputs() from mae and mse
  Florence Bockting
  committed
  12133b8
- refactor: read the source from the attribute, reuse .measure_estimate_se()
  Florence Bockting
  committed
  bd67f6a
- refactor: print the Pareto k table with pareto_k_table()
  Florence Bockting
  committed
  ef914d8
- refactor: remove dead code and correct the .compute_measure() roxygen
  Florence Bockting
  committed
  f79c4f8
Commits on Aug 31, 2026
- fix: label rps as a loss and srps as a utility in .measure_spec
  Florence Bockting
  committed
  4882727
- docs: correct the ic, mse/rmse/r2 and CRPS reference entries
  Florence Bockting
  committed
  ffa1786
- docs: correct the acc and bacc definitions in overview-measures
  Florence Bockting
  committed
  fa95ff7
- docs: document the PWM estimator that the crps and scrps code uses
  Florence Bockting
  committed
  29072e6
- docs: state that rps and srps take draws, not class probabilities
  Florence Bockting
  committed
  294f87a
- docs: correct the measure keywords and the weighted path in overview-measures
  Florence Bockting
  committed
  8a9a8e2
- docs: fit the workflow vignette's models in the article
  Florence Bockting
  committed
  6db2531
- docs: correct the pred-measure-workflow object and measure tables
  Florence Bockting
  committed
  f0715be
- docs: state the Pareto-k threshold as sample-size dependent
  Florence Bockting
  committed
  7ffedf6
- docs: drop the devtools and dplyr dependencies from the articles
  Florence Bockting
  committed
  4dd743c
- tests: shrink the pred_measure fixtures to fit the CRAN limit
  Florence Bockting
  committed
  0a3ecbb
- tests: fix the normalized-log-weights test reading a missing field
  Florence Bockting
  committed
  919b29b
- tests: drop two fixture fields that nothing reads
  Florence Bockting
  committed
  b5952b7
- Merge branch 'pred_measure' into integrate-loo_compare
  Florence Bockting
  committed
  795cd4e
- tests: shrink the model-comparison fixture
  Florence Bockting
  committed
  69346f3
- print: accept simplify on the pred_measure path
  Florence Bockting
  committed
  e52ab8e
- model_compare: accept named models in dots
  Florence Bockting
  committed
  d55d9bf
- internal: share the k-fold K-mismatch warning
  Florence Bockting
  committed
  0f830e3
- internal: drop model_compare_checks.psis_loo_ss_list()
  Florence Bockting
  committed
  769b37a
- internal: drop model_compare_matrix.psis_loo_ss_list()
  Florence Bockting
  committed
  b80c946
- docs: correct the Sivula et al. year in the glossary
  Florence Bockting
  committed
  ca8da41
- pred_measure: keep a renamed measure's own orientation
  Florence Bockting
  committed
  47af38e
- docs: remove draft scaffolding from the comparison article
  Florence Bockting
  committed
  99d71a2
- docs: let the comparison article fit its own models
  Florence Bockting
  committed
  0e07f63
- docs: point the custom-measure chunks at the new fits
  Florence Bockting
  committed
  4317ce7
- docs: rename the comparison article to model-comparison
  Florence Bockting
  committed
  0c1de0c
- docs: correct the article path in the model_compare links
  Florence Bockting
  committed
  c66e1e2
- internal: drop the vignette fits from the data generator
  Florence Bockting
  committed
  5f6bcb9
- docs: give the two remaining articles their real vignette title
  Florence Bockting
  committed
  6dbd05d
Commits on Sep 1, 2026
- model_compare: return the default object for subsampled loo
  Florence Bockting
  committed
  91bfdba
- model_compare: let simplify = FALSE follow measures
  Florence Bockting
  committed
  6621d2d
- model_compare: name the fold count and the test size
  Florence Bockting
  committed
  0944a95
- vignettes: update model-comparison tutorial with other pred_measure family types
  Florence Bockting
  committed
  0140dc1
Commits on Sep 9, 2026
- refactor: allow attribute measure_se_diff for custom measures
  Florence Bockting
  committed
  67f4d12
- docs: update developer-notes with review comments from Jonah
  Florence Bockting
  committed
  e339f48
Commits on Sep 16, 2026
- refactor: add user facade custom_measure() and remove custom_se_fn parameter from model_compare()
  Florence Bockting
  committed
  4c5a1a8
Commits on Sep 17, 2026
- refactor: remove elpd as 'base measure'
  Florence Bockting
  committed
  791ed25
- fix: update old function description and ensure correct Roxygen version
  Florence Bockting
  committed
  dc49cd1
- docs: update vignette about measure overview
  Florence Bockting
  committed
  cb93257
Commits on Sep 18, 2026
- docs: initial version of pred_measure workflow vignette
  Florence Bockting
  committed
  22d1422
- Merge branch 'loo-v3.0.0' into pred_measure
  florence-bockting
  authored
  c1d75f8
- Merge branch 'pred_measure' of https://github.com/stan-dev/loo into pred_measure
  Florence Bockting
  committed
  0e7d193
Commits on Sep 19, 2026
- Merge remote-tracking branch 'origin/pred_measure' into integrate-loo_compare
  Florence Bockting
  committed
  aa7f286
Commits on Sep 21, 2026
- docs: update developer-notes wrt custom_measure
  Florence Bockting
  committed
  6971e13
- fix: model_compare throws warning when kfold_pred_measure use different folds; update vignette example
  Florence Bockting
  committed
  8815998
- fix: pass moment_match loss into pred_measure
  Florence Bockting
  committed
  eec2140
- fix: use separate digits for measures in print method
  Florence Bockting
  committed
  dcf5fde
- fix: update snapshots
  Florence Bockting
  committed
  ba29be1
Commits on Sep 25, 2026
- refactor: remove the 'rank_by' argument for model_compare
  Florence Bockting
  committed
  c5af96b
- fix: check whether pothoc measures have been used and warn for measures other than elpd, mlpd, and ic
  Florence Bockting
  committed
  9ffff9c
Commits on Sep 28, 2026
- fix: remove diagnostic flags for measures other than elpd
  Florence Bockting
  committed
  dd2ac81
- docs: rebuild loo_pred_measure.Rd for the post-hoc correction note
  Florence Bockting
  committed
  e372858
- feat: mark and explain flipped measure signs in print
  Florence Bockting
  committed
  36795c2
- docs: move model_compare design notes to design-discussions
  Florence Bockting
  committed
  999cfa7
- docs: shorten NEWS entries for model_compare
  Florence Bockting
  committed
  de54993
- docs: update model_compare vignette
  Florence Bockting
  committed
  cd1085d
- refactor: rename argument measure to measures
  Florence Bockting
  committed
  a18b16e
- Merge remote-tracking branch 'origin/pred_measure' into integrate-loo_compare
  Florence Bockting
  committed
  954bdb1
- docs: clean up vignette 'overview measures'
  Florence Bockting
  committed
  6341145
Commits on Sep 29, 2026
- Merge branch 'pred_measure' into integrate-loo_compare
  florence-bockting
  authored
  390eeb1
- Merge branch 'loo-v3.0.0' into pred_measure
  florence-bockting
  authored
  2a77502
- Merge branch 'pred_measure' into integrate-loo_compare
  florence-bockting
  authored
  fbe6e88
- fix: remove children folder unrelated to PR
  Florence Bockting
  committed
  71b23c2
- tests: make the test fixtures smaller for CRAN
  Florence Bockting
  committed
  ee7b99b
- model_compare: rank by elpd, else by the first shared measure in alphabetical order
  Florence Bockting
  committed
  fd427d9
- model_compare: drop complexity columns by exact name, keep custom p_ measures
  Florence Bockting
  committed
  f7a7b4d
- fix: define missing fold in example of model_compare
  Florence Bockting
  committed
  a509447
- docs: remove measure-ordering code comment from example
  Florence Bockting
  committed
  102a76b
- chore: update deprecation warning to not print internal function and remove unnecessary line break
  Florence Bockting
  committed
  d03261f
Commits on Sep 30, 2026
- print.compare.loo: warn when is given for 'loo' comparisons
  Florence Bockting
  committed
  18fd34b
- docs: measures applies to all pred_measure comparisons
  Florence Bockting
  committed
  7573b3d
- docs: change title of model-comparison.Rmd and change link in function documentation in model_compare
  Florence Bockting
  committed
  07d68c8
- docs: update the description of model-compare.Rmd in 'See also' section in model_compare
  Florence Bockting
  committed
  d958069
- docs: update EVAL parameter in vignette
  Florence Bockting
  committed
  2af1cd1
- docs: update vignette section to 'Basic usage'
  Florence Bockting
  committed
  1ac844b
- fix: point custom measure errors to custom_measre()
  Florence Bockting
  committed
  fa986da
- docs: point custom measure declarations to custom_measure()
  Florence Bockting
  committed
  706bc0f
- fix: name the ranking reference when more than four measures are compared
  Florence Bockting
  committed
  88afda5
- Merge branch 'pred_measure' into integrate-loo_compare
  florence-bockting
  authored
  0bd95d6
Commits on Oct 5, 2026
- fix: remove test_that from inner test
  Florence Bockting
  committed
  c9f72c6
- fix: remove attributes names pointing to older custom_measure design
  Florence Bockting
  committed
  2041008
- Merge pull request #380 from stan-dev/integrate-loo_compare
  florence-bockting
  authored
  f26eb02
Commits on Oct 7, 2026
- fix: keep element names when pred_measure() updates a result
  Florence Bockting
  committed
  1637ede
- fix: check log_weights size when mupred is a vector
  Florence Bockting
  committed
  c3c940e
- fix: read measure_bacc() pointwise input on the stored scale
  Florence Bockting
  committed
  3328f5c
- fix: check that y in measure_acc() and measure_bacc() matches the mupred coding
  Florence Bockting
  committed
  a70f40d
- fix: accept a 3-D ylp in *_pred_measure()
  Florence Bockting
  committed
  3764847
- fix: check that predperf is a pred_measure object in pred_measure()
  Florence Bockting
  committed
  438f54f
- fix: normalize measures once in do_pred_measure()
  Florence Bockting
  committed
  1ada0a3
- docs: name only valid measures in the pred_measure input docs
  Florence Bockting
  committed
  b427aa8
- docs: state the full return contract of a custom measure
  Florence Bockting
  committed
  7002815
- docs: refer to the measures argument by its name
  Florence Bockting
  committed
  959c057
- docs: describe the matrix shape of a measure result
  Florence Bockting
  committed
  ffae11e
- docs: show the return value on each measure help page
  Florence Bockting
  committed
  988d798
- docs: remove the unused measure_metric_params block
  Florence Bockting
  committed
  db2279c
- docs: name existing helpers in the .compute_measure() docs
  Florence Bockting
  committed
  8c581a5
- docs: attach the .merge_matrix() block to its function
  Florence Bockting
  committed
  749d421
- fix: suggest a logical save_psis in the missing psis_object error
  Florence Bockting
  committed
  5481b19
- docs: move developer sections out of the migration guide
  Florence Bockting
  committed
  fea24ad
- docs: rewrite the branch NEWS entries in the release format
  Florence Bockting
  committed
  e28acd6
- update overview-measures vignette
  Florence Bockting
  committed
  e1dca92
- update overview-measures vignette
  Florence Bockting
  committed
  3fc1580
Commits on Oct 8, 2026
- docs: standardize referencing of vignettes (use always same names)
  Florence Bockting
  committed
  d627dfd
- docs: clean-up vignette
  Florence Bockting
  committed
  e05a938
- docs: create consistent references to vignettes
  Florence Bockting
  committed
  de81c46

Commits

Commits on Jul 2, 2026

Commits on Jul 3, 2026

Commits on Jul 4, 2026

Commits on Jul 6, 2026

Commits on Jul 7, 2026

Commits on Jul 8, 2026

Commits on Jul 16, 2026

Commits on Aug 12, 2026

Commits on Aug 25, 2026

Commits on Aug 27, 2026

Commits on Aug 28, 2026

Commits on Aug 29, 2026

Commits on Aug 30, 2026

Commits on Aug 31, 2026

Commits on Sep 1, 2026

Commits on Sep 9, 2026

Commits on Sep 16, 2026

Commits on Sep 17, 2026

Commits on Sep 18, 2026

Commits on Sep 19, 2026

Commits on Sep 21, 2026