test(scale): attribute physical bytes per node and edge before SCALE26

[DecisionNerd]

Problem

GraphForge scale evidence records counts and aggregate I/O, but it still cannot reconcile which artifacts and lifecycle phases own logical bytes, allocated physical bytes, retained bytes, and transient peak. Without that attribution, constant-factor ingest diagnosis and any higher-rung storage projection are unreliable.

Attempt 5 also showed why phase attribution matters: construction remained memory-bounded, while seal/publication recovery observed roughly 59 GB read and 44 GB written. Merged #971 removed several redundant authenticated payload passes, but the integrated path still needs deterministic per-phase ceilings and measured physical allocation evidence.

Objective

First implement and deterministically validate storage-owned artifact and phase attribution. Then run S20 on OVHC-AGENCY (local-linux-cgroups-v2) to populate and reconcile real host evidence. Run S22 only if S20 passes and measured resource headroom admits it. Use the adjacent completed rungs to produce a conservative S24/SCALE26 storage qualification; prohibit S24 and SCALE26 until that projection passes.

Debt / regime

• Debt type: test/proof and observability
• Quality regime: A deterministic compute plus host evidence after deterministic admission

Requirements

• Measure logical and allocated physical bytes for canonical node topology, canonical edge topology, properties, UUID membership/surrogate indexes, adjacency/CSR, catalog/manifests, construction staging/spill, portable package, and clean imported project.
• Attribute application bytes/calls to append/merge, seal_auth, shape consume/reauthentication, encode write/postwrite authentication, publication preauthentication, CAS install read/write, hydration verification, fsync/synchronization, and recovery reauthentication.
• Preserve authoritative raw byte numerators, object/block/call counts, and live-count denominators.
• Derive canonical node bytes per live node, canonical edge bytes per live edge, authoritative project bytes per live edge, and full-lifecycle peak bytes per live edge.
• Distinguish current retained bytes from transient physical peak.
• Reconcile artifact and phase totals without double-counting structurally shared or content-addressed objects.
• Use phase-boundary storage-owned counters and allocation facts; do not recursively scan the project during active ingest.
• Project higher rungs from adjacent completed rungs using the conservative measured slope and explicit headroom.

Acceptance criteria

Deterministic admission before host execution

• Schema and storage counters cover every required artifact category and application-I/O phase, with exact reconciliation rules.
• Deterministic 1x/2x/4x tests prove linear byte/call/object growth and stable normalized slopes for append, seal, publication, hydration, recovery, and retained output.
• Tests reject missing categories/phases, mismatched totals, invalid denominators, double-counted CAS/shared objects, non-reproducible ratios, and graph-size scans during active ingest.
• Logical, allocated physical, retained, and transient-peak quantities remain distinct.
• Sanitized evidence excludes graph content, paths, UUID values, cloud provider resource IDs, credentials, and secrets.

S20 host evidence (OVHC-AGENCY)

• After deterministic admission passes on the exact integrated tree, S20 on OVHC-AGENCY reports attributable raw bytes, authoritative reopened counts, phase counters, reproducible ratios, and reconciling retained/transient totals.
• Node and edge canonical costs are reported separately.
• S20 completes within its ordinary time/RSS/storage envelope on this host before any S22 attempt.

Conditional S22 and higher-rung qualification

• S22 runs only after S20 passes and measured RSS, elapsed-time, and independent local-disk headroom admit it.
• Given completed adjacent rungs, the projection records source rungs, conservative slope, projected canonical size, projected lifecycle peak, safety headroom, and qualification decision.
• No S24 or SCALE26 run occurs unless the adjacent-rung projection fits the approved safe storage envelope on OVHC-AGENCY (free capacity with 15% reserve). A one-rung projection is invalid.

BDD completion scenarios

Deterministic attribution is complete

Given equivalent 1x, 2x, and 4x construction lifecycles
When storage and authenticated phase evidence is finalized
Then every application byte/call and retained/temporary object is assigned once
And normalized slopes remain within documented linear constant-factor ceilings.

S20 storage costs reconcile

Given a completed S20 rung on OVHC-AGENCY with authoritative reopened counts
When host storage evidence is finalized
Then canonical node and edge bytes are reported separately
And all remaining retained and transient physical bytes are assigned to documented categories without double-counting.

Higher-rung projection is reproducible

Given completed adjacent S20 and S22 rungs
When the qualification projection is calculated
Then raw observations and the conservative slope reproduce the reported higher-rung peak
And S24/SCALE26 is rejected if that peak exceeds the safe host envelope.

Implementation notes

Likely surfaces include:

• storage construction/authenticated-object inventory counters
• crates/graphforge-api/tests/scale_g500_ladder.rs
• ladder and certification evidence schemas and validators
• docs/development/perf-g500-ladder.md
• the billion-edge certification runbook

Version the evidence schema if required fields invalidate existing evidence.

Observability

Emit aggregate counters only: bytes, object counts, blocks, reads/writes, seeks, fsyncs, spills, compactions, phase elapsed time, and phase-boundary allocation snapshots.

Testing

• Artifact and phase classification/reconciliation tests.
• CAS/structural-sharing deduplication tests.
• Sparse-file logical-versus-allocated tests.
• Invalid denominator, ratio, and one-rung projection tests.
• Adjacent-rung projection/refusal tests.
• 1x/2x/4x linearity and constant-factor phase ceiling tests.
• Schema fixtures and validator regressions.
• Exact-head repository gates appropriate to the changed surface.

Documentation

Document canonical size, authoritative project size, construction peak, full-lifecycle peak, application-observed I/O, and device allocation as separate quantities. Document the deterministic admission, S20, conditional S22, then S24/SCALE26 qualification sequence on OVHC-AGENCY.

Non-goals

• Compressing data before measurements identify the responsible artifact.
• Weakening correctness, integrity, durability, or filesystem admission.
• Making any single cloud volume size (e.g. historical Fly 500 GB) a universal product requirement.
• Running S24 or SCALE26 as part of this issue.
• Treating asymptotic O(N) without constant-factor evidence as sufficient.

Relationships

• Parent and close gate: feat(storage): stage topology append-only with linear ingest I/O #901
• Harness transaction-amplification sibling: fix(scale): use construction-budget Arrow batches for S20 ingest #979
• Construction prerequisites: feat(api): stream one-generation graph construction sessions #932 and fix(storage): remove redundant authenticated construction passes #971
• S20 lifecycle gate: test(scale): complete the S18-S26 lifecycle ladder on OVHC-AGENCY #900
• Final certification: test(scale): certify the S26 billion-edge round trip on OVHC-AGENCY #745

Milestone and critical-path audit (2026-09-04)

This issue contains two sequential outcomes: deterministic attribution/admission implementation, then S20 and conditional S22 host evidence. It stays open until both required outcomes are met.

• Review PR fix(scale): reconcile measured lifecycle accounting and admission #1079 against current main and this specification; its current Fixes #951 would close the issue prematurely if only implementation tests are complete. A code-only PR should reference this issue without auto-closing it until the host criteria also have evidence.
• Merge independently verified implementation fixes with normal exact-head PR gates. The issue's open state must not prevent the host execution needed to gather its own evidence.
• Once deterministic admission and the active fix(exec): plateau S18→S19 recount/query RSS so host ladder admits S20 #1094 fix pass, test(scale): complete the S18-S26 lifecycle ladder on OVHC-AGENCY #900 may execute its authorized lower rungs to supply test(scale): attribute physical bytes per node and edge before SCALE26 #951/feat(storage): stage topology append-only with linear ingest I/O #901 evidence. Their native blocked-by edges are completion dependencies, not a requirement to close test(scale): attribute physical bytes per node and edge before SCALE26 #951/feat(storage): stage topology append-only with linear ingest I/O #901 before any test(scale): complete the S18-S26 lifecycle ladder on OVHC-AGENCY #900 execution.
• Reuse test(scale): complete the S18-S26 lifecycle ladder on OVHC-AGENCY #900's single OVHC-AGENCY ladder and immutable rung evidence. Do not schedule another ladder just to close this issue.
• S24/SCALE26 remain subject to the specified adjacent-rung storage projection and 15% reserve; no acceptance criterion or admission threshold is waived.

Verified ordinary append accounting blocker (2026-09-07)

During #1094 validation on OVHC-AGENCY, a fresh ordinary public import committed and reopened successfully. A subsequent public import-session append failed validation before sealing with Validation("phase read bytes and calls disagree").

The diagnostic attribution reported inherited parent authentication read calls = 3 and shape read calls = 232 while the corresponding seal/shape read-byte values were zero. The failure reproduces without resetting process counters and with an initial edge, so it is not explained by the isolated test's counter measurement. Retained diagnostic: /tmp/gf-1094-label-encoding-append-diagnostic.log; public-facade failure: /tmp/gf-1094-label-encoding-after.log.

This falls within the existing phase-attribution acceptance criteria. Repair the pre-seal parent byte/call accounting at its source and add a deterministic public import → commit → reopen → append regression that validates progress, seals, publishes, and preserves exact counts. Keep validate_for_qualification fail-closed and preserve raw observed numerators; do not hide the failure with a skipped append or fabricated counters. The separate #1094 encoding-marker and adjacency-cache repairs do not establish this append criterion as complete.

Source diagnosis on main 51cf519e: construction initialization records inherited authentication bytes/calls and parent-catalog bytes/calls, but leaves the corresponding phase-byte fields at zero (graph_construction.rs, initial GraphConstructionEvidence). ConstructionPhaseAttribution pairs those still-zero phase bytes with the already-populated cumulative call counters. Seal later adds only receipt-validation bytes; shape computes its byte sum only when shaping completes. Repair initialization and checkpoint/resume attribution together, preserving observed numerators and assigning each read once. The existing public append regression and host-evidence acceptance criteria remain required; this source diagnosis is not validation of a fix.

Completion evidence (2026-09-09)

Completed on integrated commit 3431440425a8a4584153ed37ca7d0f3cb6f82887, using the shared #900 ladder on OVHC-AGENCY after #1167 and #1169 merged with green CI.

• The actual public SCALE6/7/8 ownership proof passed with all 15 owner views independently reconciled, complete retained/peak growth, and exact source/imported node/edge output counts. The integrated API 1x/2x/4x phase and recovery proof also passed (cargo test -p graphforge-api --test scale_g500_ladder equivalent_full_lifecycle_1x_2x_4x_has_bounded_metric_policies -- --exact --nocapture).
• S20 passed all ten lifecycle phases: 16,777,216 live edges, 450 seconds, 200,249,344-byte peak RSS, 27,191,193,600 retained bytes and 29,039,333,376-byte storage peak.
• S22 ran only after measured admission passed, then passed all ten phases: 67,108,864 live edges, 1,955 seconds, 262,193,152-byte peak RSS, 109,432,840,192 retained bytes and 116,992,155,648-byte storage peak.
• The native S20/S22 qualification adapter validated schema /4, with 24 artifact views and nine phase rows per rung. The reproducible S26 projection is 1,876,326,875,136-byte lifecycle peak, 1,749,024,768 canonical node bytes and 78,388,592,640 canonical edge bytes. It correctly refuses against 395,297,816,576 available bytes and the 141,258,578,535-byte reserve (15% of host capacity, rounded up).
• Immutable rung results, sanitized qualification and preserved scratch evidence are in the host's 900-34314404-evidence bundle. The measured workspace is empty after evidence preservation. No S24 or S26 run occurred.

This completes measurement and safe qualification, not higher-rung certification. #900, #901 and #745 remain open. The measured construction-phase peak needs reduction before the higher-rung gate can pass; merely deleting terminal sessions would not lower that already-observed peak.

[DecisionNerd]

S20 host evidence on OVHC-AGENCY is blocked on host-native ladder execution #1087 (Fly is no longer the close path).