Optimization: Replace slow DataFrame apply() iterations with vectorized operations for metadata loading

[khushthecoder]

issue: #926

Summary

This PR addresses #926 by removing multiple pandas row-wise iteration patterns (DataFrame.apply(..., axis=\"columns\") and Series.apply(lambda ...)) from metadata/cohort loading hot paths and replacing them with vectorized NumPy/Pandas operations.

The main goal is to reduce Python-level per-row overhead when working with large cohort metadata tables, which improves end-to-end load times for downstream workflows (e.g. ML feature generation and interactive dashboards).

Changes

• malariagen_data/anoph/sample_metadata.py

  • Vectorized quarter derivation from month using np.where(...).
  • Semantics preserved: month <= 0 (sentinel / missing) results in quarter == -1.

• malariagen_data/anoph/base.py

  • Vectorized unrestricted_use derivation from terms_of_use_expiry_date using boolean ops.
  • Semantics preserved:
    • NaN expiry date → True
    • ISO-string compare against date.today().isoformat() retained
    • dtype preserved as pd.BooleanDtype().

• malariagen_data/anoph/genome_features.py

  • Vectorized gene pointer glyphs and y-offset helper columns (gene_pointer, pointer_y, label_y) using np.where.
  • Semantics preserved: empty gene_label → empty pointer (no glyph displayed).

• malariagen_data/anoph/frq_base.py

  • Replaced row-wise period creation with vectorized PeriodIndex construction for year, quarter, and month grouping.
  • Reduced Python-level iteration in cohort preparation:
    • Prefer .dt.start_time / .dt.end_time when period is Period dtype.
    • Cohort label construction is fully vectorized via Series.str and concatenation.
  • Removed remaining apply(axis=1) for heatmap multi-column index label concatenation.

• Repo hygiene

  • Added .venv/ to .gitignore to prevent accidental commits of local virtualenvs.

Tests

• Ran targeted suites for the touched modules locally:
  • tests/anoph/test_frq_base.py
  • tests/anoph/test_sample_metadata.py
  • tests/anoph/test_base.py
  • tests/anoph/test_genome_features.py

Performance note

These changes remove several Python-level per-row loops in hot paths. The biggest expected wins are:

• quarter derivation and unrestricted_use computation on large metadata tables
• vectorized PeriodIndex construction for cohort grouping (removes apply(axis=1) overhead)

While the operations remain (O(N)), they now execute primarily in optimized Pandas/NumPy internals instead of repeated Python callbacks.

Maintainer note

This optimization matters because metadata and cohort tables can be very large, and row-wise .apply() scales poorly due to interpreter overhead. The approach here is intentionally conservative: keep output values/dtypes and sentinel handling identical, but replace row-wise lambdas with vectorized primitives (np.where, boolean masks, PeriodIndex, .dt accessors, and vectorized string ops).