[infra] select the evaluation suite per pipeline - #153
Merged
Conversation
The suite mechanism merged in #146 but no workflow chose a suite, so an unset EVAL_SUITE left every job evaluating every record. Pick one per pipeline: nightly runs full, pre-merge runs smoke. Forward the variable into the two containerised eval steps; the third step runs eval_cuvslam_in_docker.sh on the host and inherits it from the workflow environment. Nothing changes yet. Both suites select complete KITTI and EuRoC, so the first behavioral difference arrives with the first full-only dataset, which is what this makes possible: without it, marking a dataset full-only would have no effect and pre-merge would keep paying for nightly's coverage. PR KPI values stay comparable to the nightly history this job diffs against, because a dataset in both suites keeps one reporter config and therefore one KPI prefix. Timeouts are unchanged. A nightly x86 job measures about 30 minutes against a 120 minute budget, and smoke equals full today, so there is nothing to re-measure until a large corpus is enabled.
vikuznetsov-nvidia
requested review from
hrabeti-nvidia and
slepichev
as code owners
September 7, 2026 23:55
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (2)
💤 Files with no reviewable changes (1)
Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review. 📝 WalkthroughWalkthroughThe nightly workflow selects the full evaluation suite. The pull request workflow selects the smoke suite. Both workflows pass ChangesEvaluation suite configuration
Priority: ⬇️ Low — Defer this workflow-only change because it selects smoke versus full evaluation suites without changing current datasets, behavior, or timeouts. Estimated code review effort: 1 (Trivial) | ~5 minutes Merge Risk: ⚪ Minimal · up to Nightly runs now select the full evaluation suite while pull-request verification selects smoke; setup containers receive the same selection, with no remaining current-head risk identified. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Test Results
cuVSLAM Evaluation KPIs
Artifacts |
vikuznetsov-nvidia
enabled auto-merge (squash)
September 8, 2026 11:13
The pr-verify note also claimed smoke and full select the same records, which stops being true as soon as a full-only dataset is enabled. The constraint it was really guarding, that a dataset in both suites must keep one KPI prefix, belongs next to the registry entry that could break it rather than here.
slepichev
approved these changes
Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The suite mechanism merged in #146 but no workflow chose a suite, so an unset EVAL_SUITE left every job evaluating every record.
Pick one per pipeline: nightly runs full, pre-merge runs smoke. Forward the variable into the two containerised eval steps; the third step runs eval_cuvslam_in_docker.sh on the host and inherits it from the workflow environment.
Nothing changes yet. Both suites select complete KITTI and EuRoC, so the first behavioral difference arrives with the first full-only dataset, which is what this makes possible: without it, marking a dataset full-only would have no effect and pre-merge would keep paying for nightly's coverage.
PR KPI values stay comparable to the nightly history this job diffs against, because a dataset in both suites keeps one reporter config and therefore one KPI prefix.
Timeouts are unchanged. A nightly x86 job measures about 30 minutes against a 120 minute budget, and smoke equals full today, so there is nothing to re-measure until a large corpus is enabled.
Summary by CodeRabbit