docs: reconciliation postmortem template (closes #669)

docs/postmortems/README.md

@@ -0,0 +1,46 @@
+# Reconciliation Postmortem Usage Guide
+
+## When to Use This Template
+
+Complete a reconciliation postmortem for **every** SEV-level reconciliation failure:
+- **SEV-1**: >$10,000 discrepancy or customer-facing balance error
+- **SEV-2**: $1,000–$10,000 discrepancy or delayed settlement
+- **SEV-3**: <$1,000 discrepancy, no customer impact
+
+## Process
+
+1. **Detect**: Reconciliation job flags a discrepancy
+2. **Triage**: On-call engineer assesses severity and customer impact
+3. **Mitigate**: Apply immediate fix to stop additional discrepancies
+4. **Investigate**: Root cause analysis using the Five Whys
+5. **Resolve**: Deploy permanent fix
+6. **Document**: Complete this template within 24 hours of resolution
+7. **Review**: Postmortem review meeting within 5 business days
+
+## Template Variables
+
+The template uses `${VARIABLE}` placeholders. Fill these in with incident-specific data:
+
+| Variable | Description |
+|----------|-------------|
+| `SEVERITY` | Short severity label (e.g., "High", "Critical") |
+| `SEV_LEVEL` | SEV-1, SEV-2, or SEV-3 |
+| `DETECTION_TIME` | When the discrepancy was first noticed (ISO 8601) |
+| `RESOLUTION_TIME` | When the permanent fix was deployed |
+| `IMPACT_WINDOW` | Duration the discrepancy affected the system |
+
+## Automation
+
+Set up a `postmortem-init` script to pre-fill known fields from the incident management system:
+
+```bash
+# Pre-fill a postmortem from incident #${INCIDENT_ID}
+./scripts/postmortem-init ${INCIDENT_ID} > docs/postmortems/reconciliation-${DATE}.md
+```
+
+## Storage
+
+Save completed postmortems to `docs/postmortems/` with the naming convention:
+```
+reconciliation-YYYY-MM-DD-sev{N}.md
+```

docs/postmortems/reconciliation-template.md

@@ -0,0 +1,94 @@
+# Reconciliation Postmortem Template
+
+> **Severity**: ${SEVERITY} | **Date**: ${DATE} | **SEV**: ${SEV_LEVEL}  
+> **Owner**: ${OWNER} | **Reviewer**: ${REVIEWER}
+
+## Incident Summary
+
+**What happened?**  
+<!-- Brief description of the reconciliation failure — what ledgers, what window, what discrepancy -->
+
+**Detection time**: ${DETECTION_TIME}  
+**Resolution time**: ${RESOLUTION_TIME}  
+**Total impact window**: ${IMPACT_WINDOW}
+
+## Financial Impact
+
+| Ledger | Expected | Actual | Delta | Root Cause |
+|--------|----------|--------|-------|------------|
+| ${LEDGER_1} | ${EXPECTED_1} | ${ACTUAL_1} | ${DELTA_1} | ${CAUSE_1} |
+| ${LEDGER_2} | ${EXPECTED_2} | ${ACTUAL_2} | ${DELTA_2} | ${CAUSE_2} |
+
+**Total discrepancy**: ${TOTAL_DELTA}  
+**Affected transactions**: ${TX_COUNT}  
+**Customer-facing**: ${CUSTOMER_IMPACT}
+
+## Timeline
+
+| Time (UTC) | Event | Actor |
+|------------|-------|-------|
+| ${TIME_1} | ${EVENT_1} | ${ACTOR_1} |
+| ${TIME_2} | ${EVENT_2} | ${ACTOR_2} |
+| ${TIME_3} | ${EVENT_3} | ${ACTOR_3} |
+| ${TIME_4} | ${EVENT_4} | ${ACTOR_4} |
+| ${TIME_5} | ${EVENT_5} | ${ACTOR_5} |
+
+## Root Cause Analysis
+
+### Direct Cause
+<!-- What specifically triggered the reconciliation failure? -->
+
+### Contributing Factors
+1. **${FACTOR_1}**: ${DESC_1}
+2. **${FACTOR_2}**: ${DESC_2}
+3. **${FACTOR_3}**: ${DESC_3}
+
+### Five Whys
+1. **Why** did the reconciliation fail? → ${WHY_1}
+2. **Why** ${WHY_1}? → ${WHY_2}
+3. **Why** ${WHY_2}? → ${WHY_3}
+4. **Why** ${WHY_3}? → ${WHY_4}
+5. **Why** ${WHY_4}? → ${WHY_5}
+
+## Resolution
+
+### Immediate Fix
+<!-- What was done to stop the bleeding? -->
+
+### Permanent Fix
+<!-- What changes prevent this from recurring? -->
+
+| Action Item | Owner | Due Date | Status |
+|------------|-------|----------|--------|
+| ${ACTION_1} | ${OWNER_1} | ${DUE_1} | ${STATUS_1} |
+| ${ACTION_2} | ${OWNER_2} | ${DUE_2} | ${STATUS_2} |
+| ${ACTION_3} | ${OWNER_3} | ${DUE_3} | ${STATUS_3} |
+
+## Detection Gaps
+
+### Why didn't monitoring catch this?
+<!-- Analysis of alert coverage, thresholds, and monitoring gaps -->
+
+### New Alerts Required
+- [ ] **${ALERT_1}**: ${DESC_1}
+- [ ] **${ALERT_2}**: ${DESC_2}
+
+## Lessons Learned
+
+1. **${LESSON_1}**
+2. **${LESSON_2}**
+3. **${LESSON_3}**
+
+## Appendix
+
+### Relevant Logs
+```
+${LOG_SNIPPET}
+```
+
+### Related Incidents
+- ${RELATED_1}
+- ${RELATED_2}
+
+---
+*Template version 1.0 | Generated by postmortem automation*