Give each battery question room to finish on slow runners - #38
Open
Robbie1977 wants to merge 2 commits into
Open
Give each battery question room to finish on slow runners#38Robbie1977 wants to merge 2 commits into
Robbie1977 wants to merge 2 commits into
Conversation
The per-question cap was 240 s. Answer latency has drifted upward (mean 26 s on 11 Aug, 50 s on 19 Aug) and the tail now sits right against the cap: T4.6 finished in 239 s in run 254 while T2.6 and T3.8 were cut off at exactly 240 s. Those two are the only reason the battery gate fails, so a slow runner or a slow upstream reads as a code failure. Raise the default to 900 s, which is what run-task-battery.mjs already clamps within. The workflow_dispatch input still overrides it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Raise the task battery's per-question timeout from 240 s to 900 s, in both
places
.github/workflows/task-battery.ymlsets it: theworkflow_dispatchinput default and
TASK_BATTERY_TIMEOUT_MS. Two lines, no other change.Why
Run 254 failed with 62/64 ok and two errors, both
Timed out after 240000 ms:Run task batteryiscontinue-on-error: true, so the visible failure isFail when task battery had errors. Neither task returned a bad answer; bothwere cut off.
The cap is no longer a sane bound on answer latency. From the committed
results in
test-results/task-battery/:T4.6 finished successfully in 239 s in run 254 — one second inside the cap.
Eight of the last fifteen battery runs are red,
mainincluded, and the gatehas become a report on runner and upstream speed rather than on the code.
900 s is roughly 3.8x the slowest successful answer observed. It needs no
change to
scripts/run-task-battery.mjs, which already defaults to 600000 andclamps to [30000, 1800000]; the workflow was the only thing forcing it lower.
The job's
timeout-minutes: 90and the step's80are untouched — run 254took 17 minutes wall-clock.
How to test
This PR touches
.github/workflows/task-battery.yml, which is in theworkflow's own
paths:filter, so pushing it runs the battery against the newvalue. A green run with 64/64 ok is the check.
Follow-ups, not in this PR
eight days. The tasks that time out are the ones that fetch pages rather than
query the graph (publications, docs, "what was in the latest release"), and
run 254's log carries
TOOL FAILED | tool=get_reviewed_page. A per-fetchtimeout would stop one slow page consuming a whole question's budget.
still overwrites
latest.jsonafter a failed run.