what's the point of --hpc_project --hpc_storage ?

[reJELIN]

dear developer,

Thank you for your work.

I'm not sure to understand the interest of this two arguments: --hpc_project --hpc_storage

What's hpc_project a prefix for every outputs that will be produce ?
same for hpc_storage it is just where the pipeline can write its results if so why is there also --top_outdir ?

same --hpc_queue is a bit weird. i'm always re-configuring my nextflow.config in order for them to fit my slurm HPC, so why it's so weird here ? why can't I just replace inside your slurm.config with a mapping of the specific partition of the HPC i use ?

Best regards,
reJELIN

[a-sneddon]

Hi @reJELIN

Thanks for your question.

That's great to hear you're getting ModkitOpt up and running on your slurm system!

To clarify, --hpc_queue, --hpc_project and --hpc_storage can be specified in the nextflow.config file, or as command-line arguments when the user runs the nextflow workflow. Internally, they are used to set the corresponding nextflow directives in the profiles slurm.config, pbs.config or pbspro.config.

To clarify the difference between --hpc_storage, --hpc_project and --top_outdir:

• --hpc_storage tells nextflow which data partitions the workflow needs to have access to (for our pbspro system, an example is gdata/ab12)
• --top_outdir tells nextflow which folder to write the output results to (e.g. "modkitopt_results")
• --hpc_project tells nextflow which HPC project the workflow can schedule jobs to (for our pbspro system, an example is project ab12)

Since we didn't have access to a slurm HPC system (we use a pbspro system), we weren't able to test the slurm profile. I suspect that slurm.config may need to be modified to use the directives that are relevant for your system (such as partition as you mention). Feel free to directly modify slurm.config to fit with your system.

We would greatly appreciate community contributions for improving the slurm profile, so you're welcome to submit a pull request with your improvements :)

Please let us know if you have any further questions, and thanks for using ModkitOpt!

Thanks,
Alex

[reJELIN]

thank you for the answer, I might provide you a slurm config !

If I may at least in slurm: hpc_project seems not necessary and same for hpc_storage as in slurm you can't write or overwrite if you do not have permissions.

hpc_storage would be the equivalent of partition in slurm i suppose ?

just for clarification: will the nextflow will work if I don't provide these variables ? last time I tested it I had to "change" the nextflow code in order to make it work.

[a-sneddon]

A slurm config would be much appreciated, thank you! Thanks for your info about slurm systems as well.

At the moment, in slurm.config, hpc_project maps to account, hpc_queue maps to queue, and hpc_storage isn't used. The idea behind having hpc_queue, hpc_storage and hpc_project as parameters that get referenced in each slurm/pbs/pbspro config was so that the command-line interface of modkitopt could be consistent between HPC systems. But as you mention, the slurm config is likely not correct. We would be very happy to update this with your suggestions.

slurm.config currently:

...
queue = params.hpc_queue
account = params.hpc_project
...

At the moment, in main.nf lines 160-171 we also do input validation to check that the hpc params are provided. So to make your changes to slurm.config work for your slurm system, you may need to comment out these lines as a temporary workaround. When we update the slurm config as you suggest, we can also update the input validation in main.nf.

I hope this makes sense, please let me know if you would like anything clarified or have any further questions :)

Thanks,
Alex