Keep a finished workflow's node statuses current instead of expiring them

[romer8]

Description

Persisted node statuses expire after 60s so a stopped reporter falls back to live statuses. That is right while a workflow runs and pointless once it finishes — a workflow that left the queue with every node terminal will never report again. It is also where the fallback most often fails: finished workflows' working directories get cleaned up, and condorpy chdirs into one, so the details row raises FileNotFoundError.

Requires both the workflow and every node to be terminal. The workflow alone is not enough (a held node can be requeued); the nodes alone are not enough (a reporter that never sends a final report leaves the last node non-terminal, and that copy must keep expiring).

Changes Made to Code

• node_statuses_are_current returns true for a finished workflow whose nodes are all terminal.
• New all_node_statuses_are_terminal; a node with no status counts against the set.
• Five tests, three of which fail without the change.

[coveralls]

coverage: 99.993% (-0.007%) from 100.0% — Aquaveo:fix/trust-terminal-node-statuses into tethysplatform:main

[Copilot]

🟡 Changes recommended

Two moderate findings remain unresolved regarding held nodes and custom terminal statuses.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Updates Condor workflow status caching so completed workflows with terminal nodes retain persisted statuses beyond normal expiry.

Changes:

• Adds terminal-aware freshness logic.
• Adds five unit tests.
• Documents the fix in release notes.

File summaries

File	Summary
tethys_compute/models/condor/condor_workflow.py	Implements terminal-aware caching. Moderate (2 votes): Held may be incorrectly treated as terminal. Moderate (1 vote): custom terminal statuses are not recognized. Nit (1 vote): update reporting documentation for the exception.
tests/unit_tests/test_tethys_compute/test_models/test_CondorWorkflow.py	Adds coverage for terminal and nonterminal cache behavior.
docs/whats_new.rst	Documents the bug fix.

Review details

Suppressed comments (2)

tethys_compute/models/condor/condor_workflow.py:118

• The public reporting documentation still says that node_statuses_are_current becomes false after node_statuses_max_age and the views then read live statuses (docs/tethys_sdk/jobs.rst:324). This new terminal-workflow exception makes that contract incomplete; document that the cache remains current only when the workflow and every persisted node are terminal, while nonterminal workflows still require periodic reports.

        Age stops mattering once there is nothing left to learn. A workflow that has
        left the queue with every node in a status it cannot leave will never report
        again, so expiring its statuses only sends the views to the scheduler to be
        told the same thing -- and a finished workflow is the one whose working
        directory is most likely to have been cleaned up, which is what reading a
        node's live status needs.

tethys_compute/models/condor/condor_workflow.py:124

• This check misses custom terminal statuses: TethysJob stores those as OTH and is_terminal resolves the name from extended_properties. A CondorWorkflow reported with a registered custom terminal status will therefore keep expiring its complete node cache instead of satisfying the workflow-terminal condition. Use self.is_terminal here.

            self.cached_status in self.TERMINAL_STATUSES
            and self.all_node_statuses_are_terminal

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.