[JamieMagee]

What kind of change does this PR introduce?

Bug fix.

• PR title follows the guidelines defined in our pull request documentation

What is the current behavior?

The CLI logs repository and output failures but can still exit with status 0.

What is the new behavior (if this is a feature change)?

Return exit status 1 when a scan or result output fails, after processing the remaining repositories. Preserve the original errors and existing per-check failure behaviour. Low or inconclusive scores still exit with status 0.

• Tests for the changes have been added (for bug fixes/features)

Which issue(s) this PR fixes

Refs #5328. This fixes the skipped-repository defect and the exit-status part of the SARIF-without-policy defect.

Special notes for your reviewer

The OpenSpec delta records the exit-status and diagnostic changes. Tests cover batch continuation and a real formatter failure, with 100% statement coverage for processRepos and coverage of all changed executable lines.

The change passes strict OpenSpec validation. All 15 baseline specs pass normal validation, but strict validation reports existing requirement-length warnings in 14 unchanged specs.

Does this PR introduce a user-facing change?

Scorecard now exits nonzero when a repository scan or result output fails, while continuing the remaining scans.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.14%. Comparing base (353ed60) to head (b3cbe1e).
⚠️ Report is 420 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #5319       +/-   ##
===========================================
+ Coverage   66.80%   77.14%   +10.33%     
===========================================
  Files         230      254       +24     
  Lines       16602    14268     -2334     
===========================================
- Hits        11091    11007       -84     
+ Misses       4808     3260     -1548     
+ Partials      703        1      -702     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[justaugustus]

@JamieMagee — thanks for this!

Can you conduct an experiment for me?

I recently merged #5326, which reverse-engineered all of the observed behavior of the current codebase in OpenSpec specs, so that:

• there are some additional readable artifacts about what codebase does
• a few of us can try out spec-driven development for this repository

Some of the quirks that process found are documented in #5328 and have been correlated a few open issues/PRs, like this one.

---

The experiment:

• Rebase on latest to pull in the specs
• Ask an agent to interview you on the current behavior and what the intended change is
• Ask the agent to update the spec based on the behavior change
• Add tests for the uncovered components

---

For example, here's what the bots wanted me to write to you :)

Thanks for this, @JamieMagee! Exiting nonzero when a repository is skipped
is the behavior I'd expect too, and this lines up with #5328: it fixes
item 1 ("The CLI exits 0 when a repository is skipped") completely, and the
exit-status half of item 4 ("SARIF without --policy always fails"). SARIF
without --policy still fails, but at least the run now says so.

A few things before this can merge:

1. Spec delta. We recently landed OpenSpec baseline specs (#5326), and
   openspec/specs/cli/spec.md → "Requirement: Exit Status" currently says
   skipped repos and formatting failures leave the exit status unchanged.
   This PR changes that, so three scenarios no longer hold:

   • "Skipped repository" (--repo=::bad:: exits 0)
   • "Unknown probe skips the repository" (--local=. --probes=nope exits 0)
   • "Formatting error does not stop a multi-repo scan"

   Could you add an openspec/changes/<change-id>/ with a MODIFIED
   Exit Status requirement for cli and updated scenarios? Please also
   link #5328 in the PR description.

2. Tests. Pulling out processRepos with an injected function makes
   this easy to test, and the three scenarios above give the cases:
   unparseable repo, scan error, format error with a non-nil result, plus
   the existing runtime-error and low-score paths. codecov/patch is
   failing right now.

3. Format errors are reported as "Skipping". A formatting failure now
   prints Skipping <uri>: format results: ..., but the repo was scanned
   and its result is still checked for runtime errors, so it isn't really
   skipped. I'd keep the Failed to format results for <uri> message for
   that case, or update the spec to whatever wording we settle on.

4. Duplicate output. Each error is printed once while the scan
   continues and again in the joined error at exit. That's fine by me,
   but the spec should say which message is the canonical one.

5. --org / --repos behavior change. With this, one repo that fails
   makes the whole run exit 1. Your release note covers it, but it should
   also be stated in the spec delta, since CI users scanning many repos will
   notice.

[JamieMagee]

I gave it a go 😄

[justaugustus]

Very cool; thanks for indulging me, @JamieMagee!