Skip to content

feat: add experimental median_average_precision function#124

Closed
shntnu wants to merge 1 commit into
mainfrom
feat/median-average-precision
Closed

feat: add experimental median_average_precision function#124
shntnu wants to merge 1 commit into
mainfrom
feat/median-average-precision

Conversation

@shntnu

@shntnu shntnu commented Jan 17, 2026

Copy link
Copy Markdown
Member

Summary

  • Add median_average_precision() as an experimental alternative to mean_average_precision()
  • Uses median instead of mean for aggregating AP scores across grouped profiles

Motivation

We've observed cases where one or two outlier profiles within a group can disproportionately drag down the mean AP score, potentially masking otherwise strong phenotypic signals. Median-based aggregation may be more robust to these outliers.

Importantly, the null distribution calculation adjusts accordingly (using element-wise median of null distributions), so this should not produce optimistically biased results—the statistic is still tested in the same rigorous way as mean average precision.

This is an experimental function to explore whether median aggregation helps in these scenarios.

Details

The new function:

  • Has identical signature to mean_average_precision() for easy drop-in usage
  • Computes median of AP scores and normalized AP scores per group
  • Uses element-wise median of null distributions for p-value calculation
  • Applies the same FDR correction (Benjamini-Hochberg)

Test plan

  • Added parametrized test for median_average_precision (with/without progress bar)
  • Verified output columns exist and values are in valid ranges
  • All 64 existing tests still pass

Next steps

  • Add examples comparing mean vs median on datasets with outliers
  • Request review after examples demonstrate the behavior

🤖 Generated with Claude Code

Add a new function to compute median (instead of mean) average precision
scores for grouped profiles. This provides an alternative aggregation
method that is more robust to outliers.

Key changes:
- Add median_average_precision() in map.py with identical signature to
  mean_average_precision()
- Export the new function from copairs.map
- Add tests for the new function

The null distribution for p-value calculation uses the element-wise
median of individual null distributions, following the same pattern
as the mean-based approach.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@shntnu

shntnu commented Jan 17, 2026

Copy link
Copy Markdown
Member Author

Closing this PR based on experimental results: median aggregation doesn't improve retrieval overall (823 vs 875 significant groups, 25.0% vs 26.5%), with mixed results across annotation types.

Given the lack of clear benefit, I'd prefer to keep the API simple rather than add another aggregation option.

Details: https://github.com/broadinstitute/jump_production/issues/22#issuecomment-3764178955

@shntnu shntnu closed this Jan 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant