fasta_res_group only points to unmasked reference

[vladsavelyev]

The only place where masked reference is used is in dragen-os call, following the WARP design:
https://github.com/populationgenomics/production-pipelines/blob/main/cpg_workflows/jobs/align.py#L445

Following up on https://github.com/populationgenomics/references/pull/12/files

[lgruen]

Thanks! This looks good to me (as discussed in populationgenomics/production-pipelines#208), but we should get @katiedelange and @cassimons's approval first before merging (see https://centrepopgen.slack.com/archives/C030X7WGFCL/p1669702540107359?thread_ts=1669180382.708679&cid=C030X7WGFCL).

[cassimons]

So does this effectively makes our default reference non-configurable? We will need to use an alternate reference when we start building our RNAseq pipeline (the workflow is not alt friendly), will this prevent that configurability? Equally, once we start playing with long reads it seems people are working with ideas like aligning to a T2T reference then lifting alignments or variant calls back to hg38. Not saying we what to do this, but I always want our reference to be a configuration.

My broader point I want to ensure is that we should build our pipelines with the expectation that we will migrate references, for some projects, at some point in the future. As such (the specific version of) hg38 should always be an explicit tracible configuration.

[vladsavelyev]

The get_config()['workflow'].get('ref_fasta') was just a hack to work around the issue of mismatching md5 sums when processing the WARP-generated CRAMs with the our pipeline (it's used to read CRAMs, but not to generate: for alignment, we still hardcode the DRAGMAP masked reference).

I haven't actually thought about making the reference build version fully configurable, as I would guess other reference files apart from fasta would also be needed to be customisable. We currently maintain a full copy of the Broad hg38 bundle, and if the Broad in the future decides to build a complete new version of their bundle, we would update the bucket pointer to the whole bundle in https://github.com/populationgenomics/references accordingly.

For RNA projects, @cassimons do you think we can just use the reference files from the Broad's bundle as well and follow the WARP model? E.g. for STAR alignment, we would use indices in https://github.com/broadinstitute/warp/blob/develop/pipelines/broad/rna_seq/test_inputs/Scientific/SM-HFCB1_100ng_.65X_A3.hg38.json, and similarly for RNA-to-genome alignments, etc.

[cassimons]

No we will not use the broad indexes. They are star version specific. We need to be able to build and maintain multiple versions as required.