fix(worker): bound wedged consolidation tasks

[Sanderhoff-alt]

Summary

Consolidation worker tasks were not covered by the existing task-level wall-clock timeout. A wedged consolidation could therefore remain in processing indefinitely, hold a reserved worker slot, and prevent consolidation backlog recovery until the worker restarted. This PR extends the existing timeout path with a dedicated consolidation ceiling and keeps the behavior opt-out through configuration.

State Transition

flowchart LR
    A[Consolidation claimed] --> B{Completes before ceiling?}
    B -->|Yes| C[completed]
    B -->|No| D[Executor cancelled]
    D --> E[failed]
    E --> F[Retry API requeues task]

Loading

Configuration

Setting	Default	Semantics
HINDSIGHT_API_CONSOLIDATION_WALL_TIMEOUT	7200 seconds	Maximum wall-clock duration for one worker consolidation task
HINDSIGHT_API_CONSOLIDATION_WALL_TIMEOUT=0	Disabled	Preserve unbounded behavior explicitly

Implementation

Component	Change
Worker poller	Map consolidation to a dedicated wall-timeout configuration and reuse asyncio.timeout() cancellation/failure handling
Configuration	Add consolidation_wall_timeout with a two-hour default
Error reporting	Include the current stage and the correct environment variable in timeout failures
Tests	Cover configured timeout, disabled timeout, unknown task types, executor cancellation, and failed operation state
Documentation	Update the environment example and current/versioned configuration references

Compatibility

This is backward compatible for existing deployments: the new default only applies to consolidation tasks, uses a deliberately longer ceiling than retain, and can be disabled with 0. Existing retain and reflect timeout behavior is unchanged.

Verification

• uv run pytest tests/test_worker_wall_timeout.py tests/test_config_validation.py -q -> 98 passed
• uv run ruff check hindsight_api/config.py hindsight_api/worker/poller.py tests/test_worker_wall_timeout.py
• uv run ruff format --check hindsight_api/config.py hindsight_api/worker/poller.py tests/test_worker_wall_timeout.py

Closes #3726

[strix-security]

Strix Security Review

No security issues found.

_{Updated for 11c8a61.}

---

Reviewed by Strix
_{Re-run review · Configure security review settings}