[HOLD] [TRUNK-17917] Add failure count monitor documentation#529
[HOLD] [TRUNK-17917] Add failure count monitor documentation#529samgutentag wants to merge 2 commits intomainfrom
Conversation
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
|
Claude finished @samgutentag's task in 1m 57s —— View job PR Review
Overall this is a well-written, thorough doc. Structure, cross-links, and content are all solid. Found 4 minor issues:
|
samgutentag
changed the title
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
|
||
| For example, with a resolution timeout of 2 hours, a test that was flagged at 3:00 PM will resolve at 5:00 PM if no new failures occur. If a new failure arrives at 4:30 PM, the clock resets, and the test will not resolve until 6:30 PM. | ||
|
|
||
| Choose a resolution timeout that gives your team enough time to verify a fix has landed. A short timeout (e.g., 30 minutes) resolves quickly but may prematurely clear tests that fail intermittently. A longer timeout (e.g., 24 hours) is more conservative and ensures the test stays flagged until it has been clean for a full day. |
There was a problem hiding this comment.
we may want to flag here that the resolution timeout cannot be shorter than the detection lookback window. its maybe obvious but could trip folks up
There was a problem hiding this comment.
Great call, added in 15adbad: 'The resolution timeout must be at least as long as the detection window.'
|
|
||
| # Failure Count Monitor | ||
|
|
||
| The failure count monitor flags a test the moment it accumulates a configured number of failures on monitored branches within a rolling time window. Unlike the threshold monitor, which requires a failure *rate* calculated over many runs, the failure count monitor reacts to individual failures without needing a minimum sample size or a percentage calculation. |
There was a problem hiding this comment.
i think we're moving toward renaming "threshold monitors" to be "failure rate monitors" so that it has parity with "failure count monitors" https://trunk-io.slack.com/archives/C08AEDGMZNH/p1775066930429269?thread_ts=1775066814.530469&cid=C08AEDGMZNH
There was a problem hiding this comment.
Yep, tracking this as a separate PR. trunk2 #3459 (UI rename) is already merged and #3463 (enum rename) is open. Will do a docs pass once the enum rename lands.
☝️ notes for me to do another pass when ready!
New docs page for the failure count monitor, a new flake detection mode that flags tests as soon as they accumulate a configured number of failures on monitored branches. Updates the detection overview to include the new monitor type in the comparison table and priority matrix. Addresses review feedback: fixed repetitive wording in example walkthrough. Em dash suggestions declined per style guide (double hyphens are intentional). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
samgutentag
force-pushed
the
sam-gutentag/failure-count-monitor-docs
branch
from
474c711 to
f277770
Compare
|
Claude finished @samgutentag's task in 1m 30s —— View job PR Review
Good overall — the docs are clear, well-structured, and the cross-links all check out. Three previous inline suggestions (lines 54, 71, 75) appear to have been resolved. One formatting issue remains:
On mb1206's suggestions:
|
Resolution timeout must be at least as long as the detection window. Flagged by Meredith in PR review. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
mb1206
left a comment
mb1206
left a comment
There was a problem hiding this comment.
looks great! you may also be able to use screenshots from the preview for editing? we show how many tests have N many failures in the lookback window, and also a histogram for when tests' last failure was if thats helpful
samgutentag
changed the title
2 participants
Summary
flaky-tests/detection/failure-count-monitor.md), a detection mode that flags tests as soon as they accumulate a configured number of failures on monitored branchesflaky-tests/detection/README.md) to include failure count in the monitor types table and status priority matrixsummary.mdtable of contentsLinear tickets
Engineering authors
Context links
Files changed
flaky-tests/detection/failure-count-monitor.md(new)flaky-tests/detection/README.md(modified)summary.md(modified)Open questions
Test plan
🤖 Generated with Claude Code