probe: PROBE-POP-READOUT-1 (D-POP-1) — VERDICT KILL, and qualia is rank-inert at the frontier - #1136
Conversation
Runs the falsifier pre-registered in post-teardown-buildup-survey-v1 §6: does a population-typicality readout (an object's Cam96 distance to its subject's own basin centroid) improve the exploration frontier's ranking over the shipped `FrontierEdge::curiosity` and its MUL-weighted sibling? Whole KJV via the `bible_wave --export` seam: 70,393 triples over 31,102 verses, trained v0.1.0-cam96-data codebook, 9 split points x 25 size-preserving shuffles, 227,261 candidates pooled, deterministic, 89 s. VERDICT KILL. The readout degrades frontier precision@10 from 0.289 to 0.011 and lands below its own shuffle null (null p95 0.133). Three findings beyond the verdict: 1. The signal is real but weak: controlling for prefix frequency, partial Spearman 0.090 against a null 95th percentile of 0.020. A global trend and a useless top-k coexist because the extreme of the readout is degenerate. 2. `curiosity_gestalt` is RANK-INERT: spearman(curiosity, gestalt magnitude) = 1.000000 over 227,261 candidates under two contrasting assessments. Its magnitude is a per-graph scalar multiple of curiosity (exploration.rs:180-215), so MUL and qualia can rescale a frontier but never reorder one. 3. Plain prefix frequency reaches precision@10 = 0.756 against the shipped ranker's 0.289, and curiosity is anti-correlated with recurrence (rho ~ -0.27) — consistent with it working as designed. Any future frontier-ranking claim must clear the frequency control first. The frequency arm and the partial Spearman were a deliberate strengthening of the pre-registered design, added before the run and recorded as such. The Fisher-z and RollingFloor legs are NOT RUN: helix is not a dependency of deepnsm-v2, and Fisher-z is analytically inert under rank combination. Family 3 stays a vacancy; this narrows rather than widens it. No library surface, no type, no tenant, no ClassView, no axis set. Boards: plan §6a RESULT, EPIPHANIES entry, LATEST_STATE delta, STATUS_BOARD D-POP-1 -> Shipped (KILL), supersession index regenerated. Also records a recurrence of the destructive-prepend trap (EPIPHANIES truncated 25,172 -> 61 lines in this session's own board pass, caught by the mandatory wc -l guard and restored) in the knowledge doc that already prohibits it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PFnYKqw6d7TTiB9cT8eFdK
|
Warning Review limit reachedNext included review available in 32 minutes. View limit detailsLimit details: You’ve used the included review currently available. Your 76 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Essentials Run ID: 📒 Files selected for processing (8)
Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_be54fbe3-e118-4c86-a261-57eba1d46e02) |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d4e0ade209
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| frequency: 1.0, | ||
| confidence: *counts.get(&(c.s, c.p, c.o)).unwrap() as f32 | ||
| / (*counts.get(&(c.s, c.p, c.o)).unwrap() as f32 + 1.0), |
There was a problem hiding this comment.
For every prefix edge, this synthesizes confidence as n / (n + 1), but the inspected MassExplorer::seed_frontier creates every seeded edge with confidence 0.5 and query_count = 0. This turns A0 into an inverse-frequency ranker and produces the reported anti-correlation, whereas the actual shipped chooser initially ties all seeded edges; consequently the 0.289 baseline and A2−A0 comparison do not measure the claimed shipped behavior. Construct A0 through seed_frontier or reproduce its actual truth inputs.
Useful? React with 👍 / 👎.
| let ap: Vec<f32> = pop.iter().map(|&p| -p).collect(); | ||
| let a2: Vec<f32> = combine_ranks(&a0, &ap); | ||
| let a3: Vec<f32> = combine_ranks(&a1, &ap); |
There was a problem hiding this comment.
The pre-registered A2/A3 treatment was distance → Fisher-z → RollingFloor bucket/occupancy salience, with raw distance explicitly reserved as a disable arm, but these lines replace it with -distance and a rank mean. Although Fisher-z alone is monotone, bucket occupancy is distribution-dependent and can change candidate ordering, so the missing dependency does not make the registered treatment analytically equivalent to this substitute. The recorded KILL therefore rejects a different readout and cannot close D-POP-1 until the occupancy arm is run.
Useful? React with 👍 / 👎.
| let mean_partial_real: f32 = all_splits.iter().map(|s| s.partial_real).sum::<f32>() / n; | ||
| let mean_null_partial_p95: f32 = all_splits.iter().map(|s| s.null_partial_p95).sum::<f32>() / n; |
There was a problem hiding this comment.
The real test statistic is the mean partial correlation across splits, but this threshold averages each split's 95th percentile. In general mean(q95(X_s)) is not q95(mean(X_s)), particularly here where splits are nested and reuse the same permutation seed. Preserve each shuffle's result across splits, average by shuffle, and then take the percentile of those aggregate values; otherwise the reported null p95 of 0.020 and the resulting claim that the weak partial signal survives the null are unsupported. The A2 p@10 threshold below has the same aggregation error.
Useful? React with 👍 / 👎.
2 participants
What this PR does
Runs the falsifier pre-registered in
post-teardown-buildup-survey-v1§6 (#1135) and records its result. One commit (d4e0adeon64f8a80= main). Probe only: no library surface, no type, no tenant, no ClassView, no axis set.The question. Does a population-typicality readout — an object's Cam96 distance to its own subject's basin centroid — improve the exploration frontier's ranking over the shipped
FrontierEdge::curiosityand its MUL/qualia-weighted siblingcuriosity_gestalt?The answer: KILL. It degrades precision@10 from 0.289 to 0.011 and lands below its own shuffle null.
Setup
Whole KJV through the existing
bible_wave --exportseam: 70,393 triples over 31,102 verses, trainedv0.1.0-cam96-datacodebook (12,543 words, 12 axes), 9 split points × 25 size-preserving shuffles, 227,261 candidates pooled. Deterministic, 89 s.A candidate
(s,p,o)mined from the prefix is labelled 1 iff the exact triple recurs in the suffix.curiosity(shipped)curiosity_gestalt(two assessments)Pre-registered rule: PASS iff (a) real partial ρ > null p95 + 0.02 and (b) Δp@10 ≥ 0.05 with A2 above its null p95. (a) passes at 0.090 vs 0.020; (b) fails at −0.278 → KILL.
Three findings beyond the verdict
curiosity_gestaltis rank-inert — measured, not argued.spearman(curiosity, gestalt magnitude) = 1.000000over 227,261 candidates under two deliberately contrastingMulAssessments.exploration.rs:180-215shows why: the magnitude isbase · fw · dk · flow · trust · staunen_boost · ground_gateand every factor butbaseis per-graph. MUL and qualia can rescale a frontier; they can never reorder one. So "does population beat qualia alone" had an a-priori answer for any ranking task.curiosityis anti-correlated with recurrence — consistent with it working as designed (prefer the rare and unqueried) rather than failing. Banked rule: any future frontier-ranking claim clears the frequency control first.The frequency arm and the partial Spearman were a deliberate strengthening of the pre-registered design, added before the run and recorded as such, following
E-BASIN-WIDTH-IS-N-ARTIFACT-1(a quantity co-varying with sample count needs a partial, not a raw correlation).A survey error the run exposed
The plan named Fisher-z and
RollingFloor::occupancyas available ingredients. They are not reachable:helixis not a dependency ofdeepnsm-v2, and adding one pulls the ndarray git fork into that crate's build. Those legs are NOT RUN and say so. Under rank combination Fisher-z is analytically inert anyway.Process incident, recorded rather than hidden
This session's own board pass truncated
EPIPHANIES.mdfrom 25,172 lines to 61 with the exact prohibitedopen(p,'w').write(E + open(p).read())shape — in a script that also contained two correct prepends. The mandatorywc -lguard caught it in the same command; the file was restored from HEAD and re-prepended read-before-write, and nothing wrong was ever committed. A dated recurrence note is added to the knowledge doc that already prohibits the pattern, with the two lessons the recurrence adds.What this PR does NOT do
curiosity_gestaltis measured, not modified.Verification
cargo build,cargo clippy --example pop_readout -- -D warningsandcargo fmt --checkall clean ondeepnsm-v2(workspace-excluded, so run via--manifest-path). The probe run exits 0 and reproduces the numbers above. Board line-count guards checked againstmainfor every touched file; supersession index regenerated after the board writes.🤖 Generated with Claude Code
https://claude.ai/code/session_01PFnYKqw6d7TTiB9cT8eFdK
Generated by Claude Code