Improve parallel processing with BiocParallel

[dimalvovs]

This pull request introduces improvements to how distributed CoGAPS handles in-memory data partitioning for parallel workers, ensuring that each worker receives only the relevant subset of the data, which reduces memory usage and improves efficiency. It also adds a new parameter to control internal subsetting, updates documentation, and adds comprehensive tests to verify the new behavior.

Test results
Below are the test results of the memory footprint and execution speed (project - this branch, bau - master branch).

  kind  realtime duration    X.cpu peak_vmem cpu_seconds
1  bau  9405.125 11367.38 4.833625   82.3875    69477.08
2 proj 17132.000 17160.38 3.087625   45.7500    73279.37

Enhancements to distributed data handling:

• R/DistributedCogaps.R: Refactored the distributed execution logic so that, when data is in-memory, each worker receives only the subset it needs, and a new skipInternalSubset parameter is used to bypass redundant subsetting within the worker. [1] [2] [3]

Parallel processing:

• [R/DistributedCogaps.R(https://github.com/Improve parallel processing with BiocParallel #163/commits/3ae5dbff7e939b65a64aad6ffcf6c22a4affe71e): Now number of workers depends on available cpus rather than nSets. This avoids spawning too many workers when there are little CPUs available.

API and documentation updates:

• R/DistributedCogaps.R, man/callInternalCoGAPS.Rd: Added the skipInternalSubset argument to callInternalCoGAPS, updated function signatures, and improved documentation to reflect the new parameter and its usage. [1] [2] [3]

Testing improvements:

• tests/testthat/test_DistributedCogaps.R: Added new tests to ensure that distributed workers receive only the relevant subset of data, that memory usage is reduced, and that the new logic works for both "genome-wide" and "single-cell" modes as well as when there are more data subsets than workers.

Version update:

• DESCRIPTION: Bumped the package version to 3.33.3 and updated the release date.

[Copilot]

Pull request overview

This PR refactors distributed CoGAPS execution to reduce per-worker memory usage by subsetting in-memory data before dispatching to BiocParallel workers, and introduces a skipInternalSubset flag to avoid redundant subsetting inside workers. It also updates the internal API documentation, adds tests covering the new subsetting behavior, and bumps the package version/date.

Changes:

• Subset in-memory data/uncertainty per subset before calling callInternalCoGAPS, using skipInternalSubset=TRUE to bypass internal subsetting.
• Adjust default worker count selection to depend on available CPU cores rather than nSets.
• Add tests verifying subsetting behavior and a new “nSets > workers” scenario; update Rd docs; bump DESCRIPTION version/date.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

File	Description
R/DistributedCogaps.R	Implements per-worker in-memory subsetting and adds skipInternalSubset; adjusts default BiocParallel worker selection.
tests/testthat/test_DistributedCogaps.R	Adds tests asserting workers receive subsetted data and that runs succeed when nSets exceeds workers.
man/callInternalCoGAPS.Rd	Documents the new skipInternalSubset argument in the internal wrapper API.
DESCRIPTION	Bumps package version and updates release date.

Files not reviewed (1)

• man/callInternalCoGAPS.Rd: Generated file

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[dimalvovs]

It looks that MultiCoreParam does not do multi cores in a docker container

[dimalvovs]

SnowParam(type="SOCK"):