Consolidate bulk/single-cell TCR analysis into BULKTCR_ANALYSIS; remove dead cellranger/pseudobulk code

[dltamayo]

Summary

• Extracts a shared BULKTCR_ANALYSIS subworkflow (ANNOTATE + gated SAMPLE/PATIENT/COMPARE + gated report rendering) that both TCRTOOLKIT_BULK (bulk) and TCRTOOLKIT_SC (single-cell) now call, replacing duplicated call-graph wiring that previously lived separately in each pipeline.
• Removes bulk's --input_format cellranger code path entirely (redundant with --mode singlecell's existing pseudobulk_by_phenotype), tightening nextflow_schema.json so it's rejected upfront by VALIDATE_PARAMS().
• Follow-up cleanup: deletes a cluster of dead notebooks, plot processes (COMPARE_PLOT, GLIPH2_PLOT, SAMPLE_PLOT), an orphaned concatenation process (ANNOTATE_CONCATENATE) and its script, plus every now-unreferenced config param (10 in nextflow.config/nextflow_schema.json, 4 in Cirro launch-form config).
• Fixes all addressable nextflow lint warnings (48 → 4; the remaining 4 are confirmed linter false positives).
• Second follow-up pass (full nextflow.config/nextflow_schema.json param audit): removes dead legacy GLIPH2-CLI params and an unused GLIPH2 CLI test fixture, fixes two "shadowed" params that looked configurable but were silently hardcoded, and adds Cirro launch profiles for both single-cell routes.
• Renames the two top-level workflows for a consistent naming scheme: TCRTOOLKIT → TCRTOOLKIT_BULK (workflows/tcrtoolkit.nf → workflows/tcrtoolkit_bulk.nf), SINGLECELL_WORKFLOW → TCRTOOLKIT_SC (workflows/singlecell.nf → workflows/tcrtoolkit_sc.nf).
• Consolidates all 4 Cirro launch profiles under one .cirro/ root (.cirro/bulk_analysis/, .cirro/convert_adaptive/, .cirro/singlecell_vdjonly/, .cirro/singlecell_full_gex/), matching the convention used by a sibling BTC pipeline.
• Fixes two bugs (caught by automated PR review) in the new single-cell Cirro profiles: a samplesheet-fallback check that missed the case where a left-merge against an empty file listing still returns non-empty rows, and directory-vs-file staging for VDJ inputs on S3-backed datasets (Cirro/Nextflow can't auto-stage a bare directory string from a CSV — reworked to stage individual named files instead, adapted from WangLab-ComputationalBiology/SCRATCH-QC's pattern; the core VDJ_QC_analysis.qmd notebook is untouched, so on-prem/local runs are unaffected).

Details

Consolidation (4 tasks, each independently reviewed):

1. Rename ANNOTATE_FROM_CONCAT→ANNOTATE (now the universal shared step) and old ANNOTATE→ANNOTATE_INGEST (bulk-only raw-sample ingestion) — pure rename, verified byte-identical bodies.
2. Create subworkflows/local/bulktcr_analysis.nf, wire TCRTOOLKIT_BULK to it, remove the cellranger branch from convert.nf.
3. Wire TCRTOOLKIT_SC to the same BULKTCR_ANALYSIS engine (run_reports: false — SC has its own REPERTOIRE/MASTER_SUMMARY reports).
4. Delete the now-dead cellranger/pseudobulk-phenotype files, tighten input_format's schema enum, clean config/README.

Dead-code cleanup (found by auditing every process/subworkflow for real callers, not just imports):

• compare_calc.py/COMPARE_CALC, compare_concatenate.py/ANNOTATE_CONCATENATE, cluster_to_sc.py (superseded by enrich_seurat.R) — all confirmed zero live callers.
• COMPARE_PLOT, GLIPH2_PLOT, SAMPLE_PLOT and their notebooks — SAMPLE_PLOT was superseded by template_sample.qmd's current reporting; the other two lost their only caller when the cellranger pseudobulk-phenotype path was removed.
• Unused sample_meta emit on SAMPLE_CALC.
• Renamed bin/test.py → bin/test_tcrdist3_matrix.py (it's a manual, non-CI Python unit test, not a stray file).

Lint fixes:

• Deprecated Channel.* → channel.* (9 files), implicit it closures → explicit params, unused closure params prefixed with _.
• Removed the emit name on 3 single-output subworkflows (COMPARE, CONVERT, REPORT) and updated every call site; CONVERT was restructured onto the ternary-in-emit idiom already established in patient.nf.
• Fixed cluster_to_sc.nf's discouraged ${projectDir}/bin/... reference — enrich_seurat.R already has the same shebang/executable bit as every Python script in bin/, so it's now invoked by bare name like the rest of the pipeline's scripts.

Config/param audit follow-up (full sweep of every .config file's params against actual usage):

• Removed algorithm, global_convergence_cutoff, p_depth, ref_files — leftovers from a pre-turboGliph GLIPH2 CLI invocation, referenced nowhere in the codebase (ref_files pointed at an assets/gliph2_files directory that doesn't exist on disk).
• Fixed vdj_chain_mode: VDJ_QC hardcoded -P chain_mode="auto", silently ignoring the config param entirely (matching default value, so behavior is unchanged — this was a shadowed no-op, not a live bug).
• Fixed all_aa_interchangeable: was hardcoded FALSE in the R call and unconfigurable; now a real boolean param (default false, preserving current behavior), converted explicitly to R's TRUE/FALSE since Groovy's boolean stringifies lowercase, which isn't valid R.
• Moved GLIPH2_TURBOGLIPH's 5 tuning params from direct params.* reads inside its script block to declared process inputs, matching GIANA_CALC's existing call style in the same subworkflow (patient.nf).
• Deleted tests/test_data/gliph2-test-data/ — a GLIPH2 CLI reference/demo bundle traced to this repo's very first "add nf-test" commit, never referenced by any test or script since.
• Renamed .cirro/ → .cirro_bulk_analysis/ (it's the bulk entry point) and removed .cirro_pseudobulk/ (its only route, --input_format cellranger, is now schema-rejected). Added .cirro_singlecell_vdjonly/ and .cirro_singlecell_full_gex/ Cirro launch profiles for the two --mode singlecell routes, adapted from break-through-cancer/staple's approach to deriving per-sample directory paths from Cirro's file listing.

Workflow rename:

• workflows/tcrtoolkit.nf → workflows/tcrtoolkit_bulk.nf, workflow TCRTOOLKIT → TCRTOOLKIT_BULK.
• workflows/singlecell.nf → workflows/tcrtoolkit_sc.nf, workflow SINGLECELL_WORKFLOW → TCRTOOLKIT_SC.
• main.nf's mode dispatch updated to match; every prose reference to the old names in comments/docstrings (subworkflows/local/patient.nf, subworkflows/local/annotate.nf, subworkflows/local/bulktcr_analysis.nf, tests/main.nf.test) updated, plus a user-facing println log message in tcrtoolkit_bulk.nf that still said "Running TCRTOOLKIT workflow...".

Cirro consolidation + automated-review fixes (final round):

• Moved all 4 .cirro_* directories into one .cirro/ root with named subdirectories (bulk_analysis, convert_adaptive, singlecell_vdjonly, singlecell_full_gex), matching WangLab-ComputationalBiology/SCRATCH-QC's convention. Pure rename, git-tracked as renames.
• Fixed preprocess.py's samplesheet-fallback check in both single-cell Cirro profiles: sample_sheet.empty never caught the case where ds.files is empty but the left-merge against ds.samplesheet still returns every row (just with path as NaN) — now checks sample_sheet.empty or sample_sheet['path'].isna().all().
• Fixed VDJ input staging for Cirro/S3-backed datasets: Nextflow can't auto-stage a whole outs/ directory referenced only as a string inside a CSV. Added 3 new optional params (contigs_files, clonotypes_files, metrics_files, comma-separated and sample_sheet-ordered), a new bin/stage_vdj_files.py helper that reconstructs local outs/-shaped directories from Nextflow-staged files, and wired both into VDJ_QC/vdj_qc.nf. VDJ_QC_analysis.qmd itself is untouched — zero risk to local/on-prem execution, which never sets these params.

Intentionally left alone (flagged, not fixed):

• TCRI_SW — a fully scaffolded but never-wired single-cell feature (module + subworkflow + 65KB report notebook + run_tcri=true default), added alongside the SC integration but never connected to workflows/tcrtoolkit_sc.nf.
• d_family.csv — computed and published by SAMPLE_CALC_PIVOT but never read back downstream.
• notebooks/template_pheno_sc.qmd — orphaned; its data-loading logic doesn't match single-cell's current --by-phenotype output shape (needs a design decision, not a mechanical fix).
• Samplesheet metadata robustness — a full audit found that bulk's "optional" samplesheet columns (alias, origin, timepoint, patient) are actually treated as required by most bulk report notebooks and crash with unhandled Python errors if omitted (GH notebooks/template_qc.qmd fails without 'alias' column in samplesheet.csv #94 is one instance of this; the audit found the same pattern extends well beyond alias alone, including on the default sample,compare workflow_level). Single-cell's equivalent metadata columns are robustly guarded. Not addressed in this PR — needs a product decision on the right fix (require vs. gate-per-block vs. drop unused references) before implementation.

Test plan

• nf-test test tests/main.nf.test --verbose — all 3 cases pass against the full 35-sample fixture: "Minimal example dataset", "Single-cell VDJ-only minimal example", "Bulk mode rejects cellranger input_format" (now rejected upfront by schema validation).
• nf-test test tests/modules/ — 17/17 pass, unaffected by deleted/rewired files.
• nextflow lint . — 48 → 4 warnings (remaining 4 confirmed false positives).
• Re-verified all 3 nf-test cases pass after the config/param audit follow-up commits (6-sample fixture), including the patient-level GIANA/GLIPH2_TURBOGLIPH path exercising the new input-block wiring.
• Re-verified all 3 nf-test cases pass after the TCRTOOLKIT_BULK/TCRTOOLKIT_SC workflow rename (6-sample fixture).
• nextflow lint . re-run clean after the VDJ_QC/vdj_qc.nf staging changes (caught and fixed one new strict-syntax error: a workflow-local closure-in-def isn't callable in this Nextflow version, same workaround already used for enabled() in tcrtoolkit_sc.nf).
• Re-verified all 3 nf-test cases pass after the Cirro consolidation + S3-staging-fix commits (6-sample fixture) — the VDJ-only single-cell test exercises VDJ_QC's new staging inputs with the params left unset (local-run path), confirming no regression to the existing behavior.
• CLUSTER_TO_SC_SW/enrich_seurat.R fix is unverified by nf-test — no existing test case exercises the full-SC/GEX route (only VDJ-only is covered). Recommend a manual full-SC run before merge if GEX test data is available.
• The Cirro single-cell launch profiles, including the new file-level VDJ staging fix, are unverified against a real Cirro-hosted dataset — nf-test only exercises the params' unset (local-run) default; no existing profile in this repo has driven a directory-shaped (Cell Ranger outs/) input through Cirro before.

🤖 Generated with Claude Code

[github-actions]

Unit Test Results

17 tests  ±0   17 ✅ ±0   7m 56s ⏱️ -6s
 7 suites ±0    0 💤 ±0 
 1 files   ±0    0 ❌ ±0 

Results for commit 1602d14. ± Comparison against base commit 4ed0733.

♻️ This comment has been updated with latest results.

[Copilot]

Pull request overview

Consolidates bulk and single-cell TCR analysis into a shared subworkflow while removing obsolete Cell Ranger, plotting, and GLIPH2 code.

Changes:

• Adds shared BULKTCR_ANALYSIS orchestration.
• Removes the bulk Cell Ranger pseudobulk route and dead assets.
• Cleans configuration, lint warnings, and adds Cirro single-cell profiles.

Reviewed changes

Copilot reviewed 58 out of 72 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
workflows/tcrtoolkit.nf	Uses the shared bulk analysis engine.
workflows/singlecell.nf	Routes pseudobulk data through the shared engine.
tests/test_data/gliph2-test-data/test_data/ref_V_CD8.txt	Removes unused GLIPH2 fixture.
tests/test_data/gliph2-test-data/test_data/ref_V_CD48.txt	Removes unused GLIPH2 fixture.
tests/test_data/gliph2-test-data/test_data/ref_V_CD4.txt	Removes unused GLIPH2 fixture.
tests/test_data/gliph2-test-data/test_data/ref_L_CD8.txt	Removes unused GLIPH2 fixture.
tests/test_data/gliph2-test-data/test_data/ref_L_CD48.txt	Removes unused GLIPH2 fixture.
tests/test_data/gliph2-test-data/test_data/ref_L_CD4.txt	Removes unused GLIPH2 fixture.
tests/test_data/gliph2-test-data/test_data/demo.cfg	Removes unused GLIPH2 configuration.
tests/test_data/gliph2-test-data/test_data/demo_hla.txt	Removes unused GLIPH2 HLA data.
tests/main.nf.test	Tests rejection of bulk Cell Ranger input.
subworkflows/scratch/vdj_qc.nf	Updates channel factory syntax.
subworkflows/scratch/tcri.nf	Updates channel factory syntax.
subworkflows/scratch/tcell_integration.nf	Updates channel factory syntax.
subworkflows/scratch/repertoire.nf	Updates channel factory syntax.
subworkflows/scratch/master_summary.nf	Updates channel factory syntax.
subworkflows/scratch/consensus_clustering.nf	Updates channel factory syntax.
subworkflows/scratch/conga.nf	Updates channel factory syntax.
subworkflows/local/sample.nf	Removes obsolete sample plotting.
subworkflows/local/report.nf	Simplifies the report output.
subworkflows/local/pseudobulk_qc.nf	Applies lint and naming updates.
subworkflows/local/pseudobulk_phenotype.nf	Removes obsolete phenotype workflow.
subworkflows/local/patient.nf	Passes GLIPH2 tuning parameters explicitly.
subworkflows/local/convert.nf	Removes Cell Ranger conversion routes.
subworkflows/local/compare.nf	Simplifies the comparison output.
subworkflows/local/bulktcr_analysis.nf	Adds shared analysis and reporting orchestration.
subworkflows/local/annotate.nf	Separates ingestion from shared annotation.
subworkflows/bridges/sc_to_bulk.nf	Makes closure parameters explicit.
README.md	Documents supported bulk input formats.
notebooks/sample_stats_template.qmd	Removes superseded sample report.
notebooks/gliph2_report_template.qmd	Removes obsolete GLIPH2 report.
notebooks/compare_stats_template.qmd	Removes obsolete comparison report.
nextflow.config	Removes dead parameters and exposes active options.
nextflow_schema.json	Tightens input and parameter validation.
modules/scratch/VDJ_QC/main.nf	Honors configured VDJ chain mode.
modules/local/samplesheet/samplesheet_pheno.nf	Removes obsolete phenotype samplesheet process.
modules/local/sample/sample_plot.nf	Removes obsolete sample plotting process.
modules/local/sample/sample_calc.nf	Removes an unused output.
modules/local/report/render_notebook.nf	Updates bulk phenotype documentation.
modules/local/convert/pseudobulk_phenotype_cellranger.nf	Removes phenotype Cell Ranger conversion.
modules/local/convert/pseudobulk_cellranger.nf	Removes Cell Ranger pseudobulking.
modules/local/compare/gliph2.nf	Adds explicit GLIPH2 inputs and removes plotting.
modules/local/compare/compare_plot.nf	Removes obsolete comparison plotting.
modules/local/compare/compare_calc.nf	Removes unused comparison calculations.
modules/local/annotate/main.nf	Removes orphaned concatenation process.
modules/bridges/cluster_to_sc.nf	Invokes the bundled enrichment script via PATH.
conf/modules.config	Removes obsolete phenotype publishing rules.
bin/test_tcrdist3_matrix.py	Clarifies the renamed manual test.
bin/pseudobulk.py	Removes obsolete Cell Ranger implementation.
bin/create_pheno_samplesheet.py	Removes obsolete helper script.
bin/compare_concatenate.py	Removes unused concatenation script.
bin/compare_calc.py	Removes unused comparison script.
bin/cluster_to_sc.py	Removes superseded cluster bridge.
.cirro_singlecell_vdjonly/process-input.json	Adds VDJ-only Cirro inputs.
.cirro_singlecell_vdjonly/process-form.json	Adds the VDJ-only launch form.
.cirro_singlecell_vdjonly/process-compute.config	Adds VDJ-only compute configuration.
.cirro_singlecell_vdjonly/preprocess.py	Generates VDJ-only samplesheets.
.cirro_singlecell_full_gex/process-input.json	Adds full single-cell Cirro inputs.
.cirro_singlecell_full_gex/process-form.json	Adds the full single-cell launch form.
.cirro_singlecell_full_gex/process-compute.config	Adds full single-cell compute configuration.
.cirro_singlecell_full_gex/preprocess.py	Generates full single-cell samplesheets.
.cirro_pseudobulk/process-input.json	Removes obsolete pseudobulk profile inputs.
.cirro_pseudobulk/process-form.json	Removes obsolete pseudobulk form.
.cirro_pseudobulk/preprocess.py	Removes obsolete pseudobulk preprocessing.
.cirro_bulk_analysis/process-input.json	Removes dead bulk launch parameters.
.cirro_bulk_analysis/process-form.json	Removes dead bulk form controls.
.cirro_bulk_analysis/process-compute.config	Retains bulk compute configuration.
.cirro_bulk_analysis/preprocess.py	Defines bulk Cirro preprocessing.

Suppressed comments (2)

.cirro_singlecell_vdjonly/preprocess.py:44

• The fallback does not run when ds.files is empty but ds.samplesheet has rows: samplesheet_from_files performs a left merge, which preserves those rows and leaves path as NaN, so sample_sheet.empty is false. Check for an all-null path column (or check ds.files.empty before merging) so directory-shaped datasets actually use samplesheet_from_params.

    sample_sheet = samplesheet_from_files(ds)

    if sample_sheet.empty:
        ds.logger.warning("No files found in dataset. Preparing sample_sheet from params.")
        sample_sheet = samplesheet_from_params(ds)

.cirro_singlecell_full_gex/preprocess.py:44

• The fallback does not run when ds.files is empty but ds.samplesheet has rows: samplesheet_from_files performs a left merge, which preserves those rows and leaves path as NaN, so sample_sheet.empty is false. Check for an all-null path column (or check ds.files.empty before merging) so directory-shaped datasets actually use samplesheet_from_params.

    sample_sheet = samplesheet_from_files(ds)

    if sample_sheet.empty:
        ds.logger.warning("No files found in dataset. Preparing sample_sheet from params.")
        sample_sheet = samplesheet_from_params(ds)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[dltamayo]

Confirmed real and reachable (e.g. --workflow_level patient alone in bulk mode). Verified this is pre-existing behavior carried over byte-identical from the old tcrtoolkit.nf, not introduced by this PR. Deliberately not fixed here — tracked as item #5.1 in the outstanding-work list alongside related workflow_level validation gaps (unvalidated values, silent-success misconfiguration) so it can be fixed as one coherent change rather than a partial patch. Leaving this thread open pending that follow-up.

[dltamayo]

Fixed in f5174ce: added contigs_files/clonotypes_files/metrics_files params (comma-separated, sample_sheet-ordered file lists), a new bin/stage_vdj_files.py helper, and wired both through VDJ_QC/vdj_qc.nf so per-sample files are staged individually instead of relying on a bare directory string. This file has since moved to .cirro/singlecell_vdjonly/preprocess.py as part of the Cirro-profile consolidation (0cb49e3).

[dltamayo]

Fixed in f5174ce: added contigs_files/clonotypes_files/metrics_files params (comma-separated, sample_sheet-ordered file lists), a new bin/stage_vdj_files.py helper, and wired both through VDJ_QC/vdj_qc.nf so per-sample files are staged individually instead of relying on a bare directory string. This file has since moved to .cirro/singlecell_full_gex/preprocess.py as part of the Cirro-profile consolidation (0cb49e3).