Transient-rate escalation can never fire for a daily cron, hiding sustained failures

[raisedadead]

Summary

A cron that fails every day produces no Sentry Issue, which is why the reconcile-scheduler outage went unnoticed for 22 days.

Cause

IsTransient classifies gRPC DeadlineExceeded as transient (internal/observability/sentry.go:400-408, added fbde5c1, v1.6.0). CaptureBackground therefore downgrades it to a slog.Warn and defers to the rate tracker (internal/observability/transientrate.go:34-54), which escalates to an Issue only when:

• a given op is seen defaultTransientRateThreshold (3) times, and
• within defaultTransientResetWindow (26h) of the previous sighting, and
• at most once per process (escalated is latched and never cleared)

A daily cron sits right at the 26h boundary, and the latch plus any pod restart means escalation effectively never fires. Before v1.6.0 the same failure raised 99 Issues across v1.4.0/v1.5.0; after, zero.

Impact

The sustained-failure escalation path is dead for anything on a daily cadence — exactly the cadence of the scheduled workflows it exists to protect. Only the Sentry cron monitor caught the outage, and only because check-ins are a separate signal.

Fix direction

• Clear escalated on a fresh window rather than latching for process lifetime, or track escalation durably instead of in memory.
• Reconsider the 26h window against a 24h cron — a daily job can never accumulate 3 sightings inside it when window and period are this close.
• Consider a distinct "N consecutive failures" signal for scheduled work, independent of the transient-rate path.