Layer-3 differential: tap-retry scenario needs a fixture control that truly forces retryIfNoChange

[thymikee]

Found by the layer-3 differential's own engine-side invariant in #1289 — a case of the oracle catching a defect in itself rather than in the engine.

What happened

tap-retry-if-no-change passed on both engines (maestro=pass, agent-device=pass) but its invariant failed:

invariant violated: highest tapOn tapRetries was 0 (< 1):
a tap on an unchanging screen must re-tap; zero retries means
retryIfNoChange never ran and the scenario proved nothing

Outcome parity was green, so without the invariant this scenario would have looked like it was guarding retryIfNoChange while never once exercising it — the exact vacuity the invariant exists to prevent.

Why the current control does not work

The flow taps the fixture app's title (Agent Device Tester) on the assumption that a non-interactive element cannot change the screen, forcing the no-change retry path. But tapRetries came back 0, so our snapshot signature did change between captures. Likely cause: the title is rendered by ScreenTitle with a dynamic badge={${props.cartCount} in cart} (HomeScreen.tsx:47-50), and the home screen carries other live content — enough for the hierarchy signature to differ, so the engine sees "changed" and never retries.

What is needed

A fixture control where a tap provably changes nothing:

• a genuinely static screen region (no dynamic badge/status/async content in the subtree), or
• a dedicated inert target added to the fixture app for this purpose, or
• assert on a different observable that proves the retry path ran.

Reproducing locally (do NOT iterate via CI)

pnpm maestro:conformance:differential -- --platform ios --only tap-retry-if-no-change --trace-root .agent-device
# then inspect resultTiming.tapRetries per step:
grep replay_action_stop .agent-device/**/replay-timing.ndjson

Evidence: https://github.com/callstack/agent-device/actions/runs/29504440599

Acceptance

The scenario is marked knownDivergence in differential/scenarios.ts with this issue as its tracking link. The fix removes the declaration; the runner then fails if the scenario passes while still declared, so this cannot be quietly forgotten.

Not an engine bug: retryIfNoChange itself is unit-tested and upstream-parity-checked (2-attempt cap). This is about the device scenario being able to observe it.

[thymikee]

Correction — the diagnosis above is wrong, and the real problem is worse. Superseding it with evidence from the next run.

I reported this as vacuous (retry never fires). It is actually non-deterministic: tapRetries oscillates between runs with no change to the flow, the scenario, or the commit's device path.

Run	Result
29504440599	invariant violated: highest tapOn tapRetries was 0 (< 1)
29510020718	invariant held: tapOn tapRetries reached 1 (>= 1)

Same flow, same fixture app, same pinned Maestro. So the hierarchy signature sometimes holds still across the tap (retry fires) and sometimes doesn't (retry skipped) — consistent with the home screen's live content (the ScreenTitle cart badge and neighbours) settling differently depending on timing. It's a race, not a missing code path.

Why this matters more than the original report: a flaky scenario cannot be handled by knownDivergence at all. That mechanism assumes a divergence reproduces — a declared-but-flaky scenario flips between known-divergence (green) and stale-declaration (red) at random, i.e. a coin-flip scheduled job. That's strictly worse than having no scenario, because it teaches people to ignore the differential.

Action taken in #1289: the scenario is removed from the active differential set rather than declared. The flow stays at differential/flows/tap-retry-if-no-change.yaml and the metricAtLeast/tapRetries invariant stays implemented and unit-tested — only the device scenario is parked, so the schedule is deterministic. Device coverage for retryIfNoChange is therefore absent, tracked here.

What the fix needs: an inert control — a target whose subtree provably cannot change on tap (no badge, no async content, no animation), or a dedicated inert element added to the fixture app. Then re-add the scenario and the invariant enforces it for real. Verify determinism by running it several times locally with --only tap-retry-if-no-change before wiring it back into the schedule; a single green run proves nothing here, which is precisely the trap this issue fell into.