Improve test verification

[marvin-hansen]

A recent investigation reveled that that AI code, even with proper specs, consistently encodes narrow assumptions in tests, generates tautology tests, or adds otherwise ineffective tests.

What works though is a two stage approach

• Verify the test suite itself
• Use mutating testing to catch what the previous step may have missed.

Therefore, re-evaluate of all the crates becomes necessary:

• [ ]: deep_causality (deep_causality)
• [ ]: deep_causality_algorithms (deep_causality_algorithms)
• [ ]: deep_causality_cfd (deep_causality_cfd)
• [ ]: deep_causality_core (deep_causality_core)
• [ ]: deep_causality_data_structures (deep_causality_data_structures)
• [ ]: deep_causality_discovery (deep_causality_discovery)
• [ ]: deep_causality_ethos (deep_causality_ethos)
• [ ]: deep_causality_physics (deep_causality_physics)
• deep_causality_quantum (deep_causality_quantum) PR783
• [ ]: deep_causality_algebra (deep_causality_unified_math/deep_causality_algebra)
• [ ]: deep_causality_calculus (deep_causality_unified_math/deep_causality_calculus)
• [ ]: deep_causality_fft (deep_causality_unified_math/deep_causality_fft)
• [ ]: deep_causality_haft (deep_causality_unified_math/deep_causality_haft)
• [ ]: deep_causality_homology (deep_causality_unified_math/deep_causality_homology)
• [ ]: deep_causality_linear (deep_causality_unified_math/deep_causality_linear)
• [ ]: deep_causality_metric (deep_causality_unified_math/deep_causality_metric)
• [ ]: deep_causality_multivector (deep_causality_unified_math/deep_causality_multivector)
• [ ]: deep_causality_num (deep_causality_unified_math/deep_causality_num)
• [ ]: deep_causality_num_complex (deep_causality_unified_math/deep_causality_num_complex)
• [ ]: deep_causality_num_dual (deep_causality_unified_math/deep_causality_num_dual)
• [ ]: deep_causality_num_rational (deep_causality_unified_math/deep_causality_num_rational)
• [ ]: deep_causality_rand (deep_causality_unified_math/deep_causality_rand)
• [ ]: deep_causality_stats (deep_causality_unified_math/deep_causality_stats)
• [ ]: deep_causality_tensor (deep_causality_unified_math/deep_causality_tensor)
• [ ]: deep_causality_topology (deep_causality_unified_math/deep_causality_topology)
• [ ]: deep_causality_uncertain (deep_causality_unified_math/deep_causality_uncertain)
• deep_causality_ast (deep_causality_utils/deep_causality_ast) Closed in PR782
• deep_causality_file (deep_causality_utils/deep_causality_file) Closed in PR782
• deep_causality_par (deep_causality_utils/deep_causality_par) Closed in PR782
• [ ]: ultragraph (ultragraph)

Article on agentic tests.
https://danluu.com/agentic-testing/

[marvin-hansen]

For all math crates

Class	Raw count	Share	Scanner meaning
single-input	4,792	70.2%	No loop or table was recognized.
cherry-picked	1,222	17.9%	One input, literal expected value, and no recognized provenance.
circular	192	2.8%	Arithmetic used to construct the expected value.
tautology	73	1.1%	Only a weak predicate such as is_ok, is_some, or is_finite was recognized.
no-assertion	46	0.7%	No assertion was recognized in the function body.

Recommended order, prioritizing confirmed defects over raw scanner percentages:

1. deep_causality_topology

   • Known Hodge-star implementation failure.
   • Confirmed assertion-free test.
   • Multiple gauge, point-cloud, and gradient-flow tests that do not inspect results.
   • Largest suite and broad downstream impact.
   • Treat the Hodge-star repair as a separate architectural change from test cleanup.

2. deep_causality_uncertain

   • Confirmed ineffective sampling, seed-reset, and conditional-branch tests.
   • 92.9% single-input and 24.1% cherry-picked.
   • Small enough for a focused, high-value repair.

3. deep_causality_num_complex

   • Confirmed circular quaternion transcendental tests.
   • Highest substantial circular count: 66 candidates, 12.4%.
   • Replace copied formulas with independent reference values and algebraic identities.

4. deep_causality_tensor

   • Confirmed finite-only Gaussian-density assertions.
   • Large numerical surface with 367 single-input and 58 cherry-picked candidates.
   • Prioritize statistical extensions and decomposition/indexing kernels.

5. deep_causality_num_dual

   • Highest cherry-picked rate: 61.1%.
   • Only 90 explicit tests, making it a relatively quick mutation-testing target.

6. deep_causality_calculus

   • Small suite with high raw rates: 93.3% single-input and 43.3% cherry-picked.
   • Preserve independent analytic-derivative comparisons; investigate with targeted mutations.

7. deep_causality_linear

   • Large absolute backlog: 203 cherry-picked and 17 circular candidates.
   • Many flagged tests are actually independent exact or cross-algorithm controls, so mutation testing must drive repairs.

8. deep_causality_multivector

   • 104 cherry-picked and 17 circular candidates.
   • Focus on non-symmetric inputs, signs, grade selection, and cross-product/geometric-product identities.

9. deep_causality_num

   • Largest absolute candidate count: 255 cherry-picked tests.
   • Lower urgency because many are legitimate differential checks against standard-library operations.
   • Audit numeric kernels before trait-delegation tests.

10. deep_causality_algebra

    • Fix the vacuous f64::NAN test immediately.
    • The 22 no-assertion flags are otherwise helper, panic, or compile-time checks.

11. deep_causality_rand

    • Modest raw signal. Prioritize distribution boundaries and deterministic seeded behavior.

12. deep_causality_metric

13. deep_causality_haft

14. deep_causality_num_rational

15. deep_causality_homology

    • Mostly single-input or low-volume candidates without confirmed high-severity examples.

16. deep_causality_stats

    • Strong table/helper-driven suite; its no-assertion flags were false positives.

17. deep_causality_fft

    • Best scanner result: no tautology, circular, cherry-picked, or assertion-free candidates.

In practical batches: topology → uncertain/num_complex/tensor → num_dual/calculus → linear/multivector/num → remaining crates. Mutation survivors should determine the order within each batch.

[marvin-hansen]

For the 3 utils crates

Finding in one line

The utility suites do not reproduce the physics crate's widespread tautology, circular-oracle, or
magic-number problem. They do contain three confirmed ineffective-test defects, all in
deep_causality_file, while the high single-input count remains a risk signal rather than a defect
count.

The raw scan found no tautologies and no circular expectations. After calibration, none of its seven
cherry-picked findings was a defect. Two of its three no-assertion findings are ineffective
laziness tests; the third is a valid compile-time trait-bound test. A separate manual check found one
environment-dependent integration test that passes without exercising the loader.

Raw scanner results

Crate	Tests	Files	single-input	cherry-picked	tautology	circular	no-assertion
deep_causality_ast	24	12	23 (95.8%)	0	0	0	0
deep_causality_file	122	20	105 (86.1%)	6 (4.9%)	0	0	2 (1.6%)
deep_causality_par	9	2	7 (77.8%)	1 (11.1%)	0	0	1 (11.1%)
Total	155	34	135 (87.1%)	7 (4.5%)	0	0	3 (1.9%)

[marvin-hansen]

Scope

The audit covers:

• deep_causality
• deep_causality_algorithms
• deep_causality_cfd
• deep_causality_core
• deep_causality_data_structures
• deep_causality_discovery
• deep_causality_ethos
• deep_causality_quantum
• ultragraph

Per-crate inventory

Percentages use each crate's explicit test-function count. The final five columns are raw,
overlapping classifications.

Crate	Test files	Tests	Single-input	Cherry-picked	Tautology	Circular	No assertion
deep_causality	160	1,137	1,041 (91.6%)	90 (7.9%)	21 (1.8%)	4 (0.4%)	2 (0.2%)
deep_causality_algorithms	37	334	254 (76.0%)	23 (6.9%)	3 (0.9%)	7 (2.1%)	0
deep_causality_cfd	123	938	649 (69.2%)	129 (13.8%)	22 (2.3%)	46 (4.9%)	0
deep_causality_core	31	252	243 (96.4%)	0	0	0	0
deep_causality_data_structures	10	81	66 (81.5%)	2 (2.5%)	0	0	0
deep_causality_discovery	52	194	178 (91.8%)	5 (2.6%)	3 (1.5%)	0	3 (1.5%)
deep_causality_ethos	17	106	105 (99.1%)	0	1 (0.9%)	0	0
deep_causality_quantum	45	462	371 (80.3%)	76 (16.5%)	9 (1.9%)	3 (0.6%)	0
ultragraph	15	129	98 (76.0%)	1 (0.8%)	1 (0.8%)	2 (1.6%)	0
Total	490	3,633	3,005 (82.7%)	326 (9.0%)	60 (1.7%)	62 (1.7%)	5 (0.1%)

Recommended order of fixes

This ordering covers the nine crates in this report. It prioritizes confirmed ineffective behavior
checks, consequence of a missed defect, and then raw mutation risk; it does not sort by scanner
percentage alone.

1. deep_causality_quantum — repair the eight output-blind kernel and wrapper success tests
   first. Exact values and identity laws are available, so these are high-confidence, bounded
   fixes in a correctness-sensitive crate.
2. deep_causality — make CSM action execution observable, assert relay targets and effects,
   repair the two assertion-free tests, and replace getter-presence checks with identity checks.
   This is the largest confirmed behavioral cluster and the public root crate.
3. deep_causality_cfd — fix the momentum wrapper and projection property checks, then use
   targeted mutation testing on coupling, blackout, corridor, force, and inverse kernels. Preserve
   the independent analytic and dense-reference controls.
4. deep_causality_algorithms — pin the automatic transform downgrade numerically, then run
   mutations over BRCD scoring and Gaussian-family code. The raw circular tests should remain when
   their closed forms are independent.
5. deep_causality_discovery — make the BOSS fallback pipeline assert the produced report and
   selected outcome. Retain the match-and-panic conversion tests or express them as matches! for
   scanner clarity.
6. deep_causality_ethos — make successful inheritance linking observable through graph or
   verdict behavior. This is one localized, inexpensive repair.
7. ultragraph — assert that topological sorting an empty graph returns an empty ordering, then
   mutation-test centrality normalization before changing its independently derived expectations.
8. deep_causality_data_structures — no confirmed hard defect; use focused index mutations as
   a verification pass after the higher-risk crates.
9. deep_causality_core — no hard candidate from this audit. Review only through mutation
   survivors; do not expand tests merely to reduce the single-input percentage.

[marvin-hansen]

The physics crate is particularly bad:

Class	Count	Share	What it means
single-input	1616	91.9%	One hand-chosen input. No loop, no table, no range.
cherry-picked	641	36.5%	One input, a magic literal expectation, and no stated provenance for it.
tautology	169	9.6%	No assertion says anything about the answer — typically assert!(x.is_ok()) and nothing else.
circular	57	3.2%	The expected value is recomputed in the test body from the same formula the kernel uses.
no-assertion	1	0.1%	No assertion, and no delegation to a helper that asserts.

The classes overlap: a cherry-picked test is also single-input by construction.