Summary
Two reconcile workflow runs have been in RUNNING state since 2026-07-15 — three weeks — in the production Hatchet run history. Nothing reaps them.
Detail
From v1_statuses_olap: reconcile shows 985 COMPLETED, 3 FAILED, and 2 RUNNING with min(inserted_at) = max(inserted_at) = 2026-07-15.
Needs triage to establish whether this is:
- an artemis-side handler that can block indefinitely (
ReconcileSite has no explicit timeout of its own — it inherits whatever ctx Hatchet supplies), or
- upstream Hatchet behaviour where an orphaned run is never transitioned after a worker disappears.
Why it matters
Orphaned RUNNING rows distort any "is the fleet healthy" read and, if they hold a concurrency slot keyed on site, could starve later runs for the same site.
Fix direction
Establish the cause first. If artemis-side, bound ReconcileSite with an explicit deadline. If upstream, confirm whether the deployed engine version reaps stale runs and track the upstream issue.
Summary
Two
reconcileworkflow runs have been inRUNNINGstate since 2026-07-15 — three weeks — in the production Hatchet run history. Nothing reaps them.Detail
From
v1_statuses_olap:reconcileshows 985 COMPLETED, 3 FAILED, and 2 RUNNING withmin(inserted_at) = max(inserted_at) = 2026-07-15.Needs triage to establish whether this is:
ReconcileSitehas no explicit timeout of its own — it inherits whatever ctx Hatchet supplies), orWhy it matters
Orphaned RUNNING rows distort any "is the fleet healthy" read and, if they hold a concurrency slot keyed on site, could starve later runs for the same site.
Fix direction
Establish the cause first. If artemis-side, bound
ReconcileSitewith an explicit deadline. If upstream, confirm whether the deployed engine version reaps stale runs and track the upstream issue.