Skip to content

Requeue LSF jobs that fail at launch with exit 255 - #5

Open
rdemko2332 wants to merge 2 commits into
masterfrom
exclude_255_fix
Open

rdemko2332 wants to merge 2 commits into
masterfrom
exclude_255_fix

Conversation

@rdemko2332

Copy link
Copy Markdown
Contributor

During the GenomicsDB rebuild02 run on PMACS LSF (queue rhel9), some LSF jobs died while LSF was starting them: exit 255, 0s run time, no output. The job script never runs, so .command.begin/.command.exitcode are never written, Nextflow keeps the task SUBMITTED forever, and the run hangs. This stuck 27 of 30 controllers within half a day.

bsub -Q "EXCLUDE(255)" tells LSF to requeue a job that exits 255 under the same job ID on another host instead of abandoning it.

Mirrors the fix already merged in ngs-samples-nextflow, bulk-rnaseq-nextflow, and dnaseq-nextflow.

Note: I couldn't find a call site for this repo in ApiCommonWorkflow's XML at all — it may be invoked from a different top-level workflow repo. Adding the standard conf/lsf.config for consistency with the other pipelines regardless.

Jobs that die while LSF starts them never write .command.begin, so Nextflow
keeps them SUBMITTED forever and the run hangs. bsub -Q "EXCLUDE(255)" makes
LSF requeue them under the same job ID on another host.

This repo had no conf/lsf.config at all; adding the standard one used by the
other VEuPathDB nextflow pipelines so LSF runs get the same protection.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017MTiUPAZXDAZe8iw71XG4v
@wbazant

wbazant commented Nov 13, 2026

Copy link
Copy Markdown

Try MicrobiomeWorkflow instead of ApiCommonWorkflow :). Nice to see this is still running.

Move per-process memory/retry logic out of conf/lsf.config and into a new
conf/processResources.conf, expressed with Nextflow's native `memory =`
directive instead of bsub -M/-R strings, so it applies unchanged under any
executor. conf/lsf.config now holds only LSF-specific submission bits
(executor, -Q EXCLUDE(255)). This is prep for moving off LSF to Slurm or
another compute service.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017MTiUPAZXDAZe8iw71XG4v
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants