Implement LeverageSHAP approximator - #524
Conversation
Forward-looking spec for the 3 new SV approximators (LeverageSHAP, PolySHAP, OddSHAP). Approximator classes are looked up dynamically by name, so the file auto-skips classes that have not yet been registered in shapiq.approximator. As each implementation lands, the corresponding parametrizations activate. - Interface conformance (always required): index='SV', n_players, max_order/min_order, values shape and dtype, interaction_lookup. - Numerical convergence vs ExactComputer (xfail strict=False): atol schedule by budget percentage. - Determinism: same (n, random_state, budget, game) -> bit-identical output. 75 tests, all currently SKIP on main. Will activate as classes land.
Honors the cross-method testing platform promised to the tutor:
unified harness covering every SV approximator in shapiq (the
existing 11 — KernelSHAP, SVARM, Permutation*, ProxySPEX, ... — and
the 3 new ones from this project) instead of only the new line-up.
Approximator list is sourced dynamically from
shapiq.approximator.SV_APPROXIMATORS (canonical registry) plus the
3 new project names, deduplicated. Future shapiq additions land in
the harness automatically.
Split into two scopes:
* test_interface_conformance — strict shape/dtype/index/lookup
contract from the API spec. Applied ONLY to the 3 new
approximators (the contract is ours; existing methods have
different default output conventions like ProxySPEX defaulting
to FBII and max_order=n).
* test_numerical_convergence_vs_exact + test_determinism — apply
to ALL SV approximators. Cross-method comparison against
ExactComputer ground truth on identical SOUM games. xfail with
strict=False so methods that do converge surface as XPASS;
methods still under development surface as XFAIL.
Two robustness helpers:
* _construct_or_skip — tries (n=, index='SV', max_order=1,
random_state=) first (covers multi-index methods like SPEX,
ProxySPEX, ProxySHAP, MSRBiased, kADDSHAP), then falls back to
minimal signature for SV-only methods (KernelSHAP, OwenSamplingSV).
* _safe_approximate — skips on ValueError raised by approximators
that explicitly refuse a regime (e.g. SPEX 'Insufficient budget
to compute the transform' at low budgets).
Results: 10 passed, 95 skipped, 90 xfailed, 23 xpassed. The 23
xpassed are existing shapiq SV methods that converge cleanly at
full budget on small SOUM — a useful baseline for the upcoming
benchmark report.
Drop-in framework that any teammate can merge into their feature branch
to run head-to-head benchmarks against ExactComputer across every SV
approximator in shapiq, then plot the standard SHAP-literature metric
curves. No source files are modified — adds a top-level benchmark/
package, a single test file, and a small in-place test-helper sys.path
hook. Does not touch pyproject.toml or any other upstream config.
Files added:
* benchmark/__init__.py: makes the runner a proper Python package so
invocation is 'python -m benchmark.performance'.
* benchmark/_discovery.py: single source of truth for SV approximator
discovery + SV-mode construction. Holds:
- PROJECT_APPROXIMATOR_NAMES: LeverageSHAP, PolySHAP,
PolySHAPKAdd / Partial / Prior, OddSHAP.
- _SV_CONSTRUCT_OVERRIDES: per-class kwargs for non-standard
constructors (PolySHAP variants need max_order /
n_explanation_terms / q_prior).
- construct_for_sv(): three-stage construction (override ->
explicit SV signature -> minimal signature), returning
(estimator, exc) so the caller can report the most informative
exception. A ValueError from inside a matched signature wins
over a TypeError from a signature mismatch.
- safe_approximate(): catches ValueError and RuntimeError so
sparse approximators that refuse a budget regime (SPEX,
ProxySPEX, ...) skip the cell cleanly instead of crashing.
* benchmark/performance.py: CLI runner that consumes _discovery,
sweeps (method, game, budget, seed), records every cell in a
long-format CSV, and emits one PNG per (game, metric) plus a
runtime PNG. Seven metrics chosen from the union of LeverageSHAP,
PolySHAP, OddSHAP and shapiq.benchmark.metrics literature:
MSE / MAE / SSE / SAE / Precision@5 / Precision@10 / KendallTau.
Includes a '--check' interface-probe mode that prints a
constructibility table without running a sweep.
* benchmark/README.md: usage doc covering merge workflow, --check,
sweep CLI, output layout, CSV format, metric definitions, plot
conventions, and notes on the multi-index approximators that need
explicit (index='SV', max_order=1).
Files modified:
* tests/shapiq/tests_unit/tests_approximators/test_approximators_vs_exact.py:
now imports the shared helpers from benchmark._discovery via a
tightly-scoped sys.path hook at the top of the file. Picks up the
ValueError-priority construction and the RuntimeError-catch that
the test file previously did not have. Interface conformance is
now applied to the project's six new approximator names
(LeverageSHAP, PolySHAP + 3 variants, OddSHAP), so Matthias's
PolySHAP variants are no longer silently skipped by the contract
check.
Verified locally:
* pytest test_approximators_vs_exact.py: 10 passed, 170 skipped,
87 xfailed, 26 xpassed. No failures.
* python -m benchmark.performance --check: surfaces all 17 method
names (11 existing on main + 6 project additions) correctly.
* Drop-in compatibility verified by temporary merge into all three
feature branches (oddshap_approximator, leverageSHAP, PolySHAP) —
clean merge in each, --check picks up the local approximator.
|
Finally all tests pass again after:
========================================== 1270 passed, 166 skipped, 124 xfailed, 32 xpassed, 5181 warnings in 956.40s (0:15:56) ==========================================
real 16m2,138s
user 119m51,705s
sys 0m37,665s |
…tions set, Z_list and probs_list and create all-true and all-false coalitions
…x pre-commit errors
…determinism on LeverageSHAP()
…ferent game variables to avoid access counters interfering; Also compare metadata
…ames produce (slightly) different outputs
…d tiny-n edge case and add comments to document and explain the test
…use its core claim was not reliable With n = 6, a budget of 100 is above 2^n = 64, so the implementation enters the full-budget/exact regime. In that regime, the result should be identical no matter which seed you use, so asserting that different seeds must differ is false and will fail even though the code is correct. => I lowered the budget to budget=20
…o test_exact_regime_seed_independence and test_stochastic_regime_seed_variability
…stochastic regime
…est_exact_matches_multiple_small_games, test_null_player_axiom and test_minimal_budget_sweep
…her n to avoid minimal floating errors
…to base regression class
…want to make wrong assumptions)
|
@Advueu963 As requested in the last meeting, I have updated and expanded the benchmark notebook to evaluate our custom LeverageSHAP implementation (Algorithm 2) against the uniform KernelSHAP baseline across 25 configurations and 4 distinct datasets. The results demonstrate that our method surpasses KernelSHAP + np.zeros, with the custom implementation achieving a lower absolute L2 error and winning in 22 to 28 out of 30 budget steps across almost every configuration (please see the table below). Also, our Bernoulli sampling mechanism resolves the "zig-zag" pattern seen in 9 out of 25 baseline plots for KernelSHAP + np.zeros. I assume the "zig-zag" occurs when the KernelSHAP + np.zeros setup runs out of budget mid-layer. Some configurations show extreme negative percentages in the relative "Avg Improvement" column. I think this caused when the baseline randomly hits a near-zero error in its symmetry valleys which then distorts the relative average. The fully cleaned and documented notebook is now ready for review. I also added a "Empirical Evaluation: Custom LeverageSHAP vs. Uniform Weighting Baseline" section which describes and tries to explain the observations. We also plot every single experiment (i.e. each line in the table below in the Jupyter NB). 
You can also find the Jupyter NB exported in the PDF here: |
…NBs will not be pushed to the shapiq repository according to the last meeting)
|
@Advueu963 I moved the notebooks located in |
mmschlk
left a comment
mmschlk
left a comment
There was a problem hiding this comment.
Nice work! I only really have one issue with the static method of the solve_regression. I do not like that you re-export the function without any purpose. Delete this and then I also think this PR is good to go.
The discussion with the different sampling strategies is also very detailed!
| def solve_regression( | ||
| self, | ||
| X: np.ndarray, | ||
| y: np.ndarray, | ||
| kernel_weights: FloatVector, | ||
| *, | ||
| use_svd: bool = False, | ||
| ) -> np.ndarray: | ||
| """Solve the weighted regression system used by regression approximators. | ||
|
|
||
| Args: | ||
| X: The regression matrix of shape ``[n_coalitions, n_interactions]``. | ||
| y: The response vector for each coalition of shape ``[n_coalitions]``. | ||
| kernel_weights: The weights for the regression problem for each coalition. | ||
| use_svd: If ``True``, solve via the SVD-backed least-squares path directly. | ||
|
|
||
| Returns: | ||
| The solution to the regression problem. | ||
| """ | ||
| return solve_regression(X=X, y=y, kernel_weights=kernel_weights, use_svd=use_svd) | ||
|
|
There was a problem hiding this comment.
I do not like that you added a new static wrapper function to the real implementation in the module. While, yes having the function on the class level is a nice quality of life upgrade but also adds another maintenance burden which can rot over time. This is already the case haha as your docstring here is already out sync with the docstring in the real method (the docstring omits the SVD-fallback explanation and the [n_interactions] return shape that the module function documents). You are also not overriding the implementation here or any subclasses, so the addition is purely cosemetical.
If you really need to include it because of some thing I did not spot then the more honest way would be by bounding the variable to the staticmetho directly:
class Regression:
solve_regression = staticmethod(solve_regression)There was a problem hiding this comment.
My call: just delete this re-export and use the module-level function. :)
There was a problem hiding this comment.
Very good catch! I refactored the code according to your feedback and deleted the solve_regression() re-export in 23d8de4. We now use direct module-level function calls instead.
If possible, before merging allow me to quickly review all doc strings again and check the code for possible code smells. I wanna make sure the code is 100 % perfect before merging. I'll then also answer your question below and add the answer to the doc string in leverageshap.py so other users don't wonder why we use our own implementation instead of Coalition Sampler.
I'll notify you when I'm confident we're 100 % ready to merge.
There was a problem hiding this comment.
No rush. We can do this easily after the meetings! :)
There was a problem hiding this comment.
Yes, I like this evaluation. Can you tell me why your sampling is beating the Coalition Sampler exactly? Could you find out the root cause?
Thank you very much, I'll look at the feedback, changes, question, etc. now. 👍 |
|
(GitHub's UI doesn't allow replying directly below that comment so I comment here)
This one was a bit hard and it took me a while. But I think I might have found the root cause. I reproduced it in a Jupyter NB in this commit on our submission Draft PR on our own fork: 53900ed Please look at this line in our custom sampler: prob = 1.0 if full_count <= two_c else two_c / full_countIf, for a layer with size s the total number of all possible combinations For example: With a budget of m=100 and n=8, the parameter 2c is large enough that the layers s=1 (single players) and s=2 (pairs), as well as their complements s=7 and s=6, are fully enumerated. This is better because the layers s = 1 and s = 2 contain the most important information about the main effects and pairwise interactions of the XGBoost model. Because our sampler captures them completely and deterministically, the sampling variance for these essential layers is exactly 0. Coalition sampler selects coalitions in these small layers probabilistically. And that carries the risk of completely overlooking important single or pairwise coalitions or selecting them redundantly multiple times. I reproduced this in the mentioned Jupyter NB: Please look at this plot: As you can see LeverageSHAP w/o Bernoulli and LeverageSHAP w/o 2c limit are nearly identical. In the NB I computed the L2 error for each method and computed the avg. win rate: Summarized: A win rate of 61,7 % doesn't sound too good but keep in mind we are comparing two already highly optimized sampling variants here. If we include KernelSHAP as a base-line/control you can see LeverageSHAP is generally performing very well (as expected): So I believe the main "leverage" for our LeverageSHAP custom sampler implementation vs. CoalitionSampler is the calculation of What do you think? |
|
The PR is now 100 % ready and we're confident that it can be merged as-is. 👍🎉 |
Motivation and Context
In this PR we added the implementation of the LeverageSHAP approximator, based on the paper by Musco & Witter (2024). We built (1) a custom sampler for leverage score sampling (uniform size + paired sampling) (2) implemented the regression solver using the row-centering trick (Lemma 3.1) and (3) added different test cases. We originally opened a PR in our fork (FabianK-Dev#1) which I closed now and I am reopening here.
A few disclaimers / important notes:
src/shapiq/approximator/regression/our_impl_progress.md(as requsted in the project description) but this file is stil work-in-progress and not really ready for review, yet.Public API Changes
Details: Added LeverageSHAP class to
shapiq.approximator.regression.How Has This Been Tested?
We added many unittests to cover the following things:
lstsqsolver maintains the efficiency axiom on ill-conditioned matrices.You can run all new unittests using:
uv run pytest tests/shapiq/tests_unit/tests_approximators/test_approximator_leverageshap.pyTests are passing:
Checklist
We haven't completed all points on the checklist yet, as this is still an early PR where we ask for feedback but don't plan to merge it into the upstream repository, yet.
Documentation has been updated (if the public API or usage changes).An entry has been added toCHANGELOG.md(if relevant for users).