semaphore-test-reports-rspec

An RSpec formatter for Semaphore test reporting. It writes two carriers from one run: a semaphore/test-report JSON Lines stream (the native format) and enriched JUnit XML (for every consumer that still speaks JUnit). Both carry what the stock JUnit formatters drop:

• One record per attempt. With rspec-retry, every failed attempt is reported with its own failure payload, followed by the final result — so a test that failed once and recovered is visible as exactly that, not as a clean pass.
• Identity that survives renames and duplicates: file and line on every record, next to the display name.
• Failure taxonomy: assertion failures (RSpec::Expectations::*) become <failure>, everything else becomes <error> — a harness problem is distinguishable from a failing test.
• Real execution timestamps per test, and the suite's randomization seed — order-dependent flakes become reproducible.
• User metadata: scalar RSpec tags (it "pays", owner: :payments) are exported as namespaced properties.

Zero runtime dependencies. The gem depends on nothing but RSpec itself. rspec-retry is detected when present, never required.

Output

--out report.xml produces both files:

File	Format
report.xml	JUnit XML, one <testcase> per attempt
report.semaphore.jsonl	semaphore/test-report 2.0-draft.2

The .semaphore.jsonl path is the --out path with .xml replaced; set SEMAPHORE_TEST_REPORTS_JSON to put it somewhere else. Without --out (streaming to stdout) only the XML is written.

The stream is written as the suite runs — header on start, one line per test as it finishes, trailer at the end — and flushed per line. A run killed mid-suite therefore leaves a valid prefix with no trailer, which is exactly how a consumer detects truncation.

Install

# Gemfile
group :test do
  gem "semaphore-test-reports-rspec"
end

Use

# .rspec
--require semaphore/test_reports/rspec
--format Semaphore::TestReports::RspecFormatter
--out report.xml

Keep your human-facing formatter alongside it (--format progress). Then publish as usual:

epilogue:
  always:
    commands:
      - test-results publish report.xml

What the output looks like

report.semaphore.jsonl — one self-contained JSON object per line (wrapped here for reading; the real lines are not):

{"kind":"report","schema":"semaphore/test-report","version":"2.0-draft.2",
 "reporter":"semaphore-test-reports-rspec 0.1.0","framework":"rspec",
 "hostname":"job-42a1","seed":18293,"started_at":"2026-08-24T12:01:02.701+00:00"}
{"kind":"test","identity":{"name":"Flaky recovers","classname":"spec.flaky_spec",
 "file":"spec/flaky_spec.rb","line":7},"state":"passed","duration_ms":412.331,"tags":{},
 "attempts":[
   {"index":0,"state":"failed","kind":"framework_retry",
    "failure":{"type":"RSpec::Expectations::ExpectationNotMetError",
               "message":"expected true, got false","stack":"…","truncated":false}},
   {"index":1,"state":"passed","kind":"framework_retry",
    "duration_ms":412.331,"started_at":"2026-08-24T12:01:03.114+00:00"}]}
{"kind":"summary","tests":1,"complete":true,"duration_ms":460.02}

report.xml — the same run, one <testcase> per attempt:

<testcase name="Flaky recovers" classname="spec.flaky_spec" file="spec/flaky_spec.rb" line="7">
  <properties>
    <property name="semaphore.file" value="spec/flaky_spec.rb"/>
    <property name="semaphore.line" value="7"/>
    <property name="semaphore.seed" value="18293"/>
    <property name="semaphore.attempt" value="0"/>
    <property name="semaphore.attempts" value="2"/>
    <property name="semaphore.attempt_kind" value="framework_retry"/>
  </properties>
  <failure message="expected true, got false" type="RSpec::Expectations::ExpectationNotMetError">…</failure>
</testcase>
<testcase name="Flaky recovers" classname="spec.flaky_spec" file="spec/flaky_spec.rb" line="7"
          time="0.412331" timestamp="2026-08-24T12:01:03.114Z">
  <properties>…<property name="semaphore.attempt" value="1"/>…</properties>
</testcase>

Development

bundle install
bundle exec rspec

The integration suite runs the formatter against fixture suites in a subprocess and asserts on both emitted carriers. Every line of every emitted stream is validated against a vendored copy of the format schema — see spec/fixtures/SCHEMA-SOURCE.md for how to re-sync it. The schema validator (json_schemer) is a development dependency only; it is never loaded at runtime.

License

Apache-2.0