scrollUntilVisible: agent-device fails to find an element real Maestro finds (layer-3 differential)

[thymikee]

Found by the layer-3 conformance differential on its first working run (#1289) — real Maestro and agent-device executing the same flow on the same simulator, same app, same pinned Maestro 2.5.1.

The divergence

Flow: scripts/maestro-conformance/differential/flows/settle-after-tap.yaml, against the fixture app com.callstack.agentdevicelab:

- launchApp:
    clearState: true
- assertVisible: Agent Device Tester
- scrollUntilVisible:
    element:
      id: home-open-form
- tapOn:
    id: home-open-form

Engine	Outcome
Maestro 2.5.1	pass (exit 0) — scrolls, finds home-open-form, taps it
agent-device	fail (exit 1)

Replay failed at step 3 (scrollUntilVisible "home-open-form"):
Maestro scrollUntilVisible target did not become visible.
(settle-after-tap.yaml:line 11)

home-open-form is a real control on the fixture app's home screen (HomeScreen.tsx:89), inside a ScrollView, below the fold — the app's own helpers/open-checkout-form.yaml scrolls it into view the same way. Maestro reaches it; we time out.

This is engine correctness in an advertised command (scroll and scrollUntilVisible is in the supported subset), not a documented deviation.

Evidence

Run: https://github.com/callstack/agent-device/actions/runs/29504440599 (differential-report.json artifact attached to the run)

{
  "id": "settle-after-tap",
  "maestro":     { "outcome": "pass", "exitCode": 0 },
  "agentDevice": { "outcome": "fail", "exitCode": 1 },
  "outcomeDiverged": true
}

The other three scenarios in the same run rule out environment as the cause — percent-swipe and optional-warned-not-failed were pass/pass on both engines.

Leads (untested hypotheses, ranked)

1. visibilityPercentage === 100 boundary — first bet. Our preset is scrollUntilVisiblePercentage: 100 (compatibility-policy.ts). If upstream's visibility threshold accepts partial visibility at the default while we demand the element be fully visible, Maestro finds it at ~97% visible and we keep scrolling until the timeout — exactly the observed shape (a timeout, not a miss).
2. speed→duration mapping. Upstream trunc(1000·(100−speed)/100)+1 (mirrored in maestroScrollDurationFromSpeed) vs our scroll-step geometry — a step size/direction mismatch could overscroll past the target.
3. Settle interplay after each scroll step — if we settle differently between steps, the visibility check may sample at the wrong moment.

Reproducing locally (do NOT iterate via CI)

The differential runner takes --only, so this loops in minutes on a local sim rather than 40-minute CI cycles:

pnpm test-app:install && pnpm --dir examples/test-app exec expo run:ios --configuration Release
pnpm maestro:conformance:differential -- --platform ios --only settle-after-tap --trace-root .agent-device

Acceptance

The oracle itself is the acceptance test. The scenario is currently marked knownDivergence in differential/scenarios.ts with this issue as its tracking link, so the scheduled differential stays green while it is a known, declared gap. The fix PR removes the knownDivergence declaration — the differential then enforces that it stays fixed, and the runner fails if a declared divergence starts passing, so a stale declaration cannot linger.

Note this also currently blocks bug class 4's device detector: settle-after-tap is the settle-ordering scenario, and it cannot reach its tapOn step while scrollUntilVisible fails.

[thymikee]

Live evidence from the differential, and it redirects the investigation. My original ranked hypotheses were partly wrong — recording that before someone spends time on them.

Both engines, same flow, same simulator (iPhone 17 Pro, iOS 26.2)

Real Maestro completed the scroll with the same parameters we use:

Scrolling DOWN until id: home-open-form is visible with speed 40,
visibility percentage 100%, timeout 20000 ms, with centering disabled... COMPLETED
Tap on id: home-open-form... COMPLETED
Assert that "Checkout form" is visible... COMPLETED

agent-device, same flow:

Replay failed at step 3 (scrollUntilVisible "home-open-form"):
Maestro scrollUntilVisible target did not become visible.

This weakens hypothesis 1 (the visibilityPercentage === 100 boundary)

I ranked that first, reasoning that Maestro might accept partial visibility where we demand 100%. Maestro's own log says it ran at visibility percentage 100%, speed 40, timeout 20000 ms and still found the element. Same knobs, same values, different outcome — so the divergence is unlikely to be the threshold itself. Worth re-checking whether our 100% check means the same thing as upstream's (element bounds fully inside the viewport vs. inside the scroll container, say), but it is no longer the obvious first suspect.

Stronger lead: which container are we scrolling?

From our failure diagnostics on the same screen:

Screen: 20 actionable ref(s) captured (refsGeneration 816368).
  @e1 [application] "Agent Device Tester"
  @e2 [window]
  @e4 [other] "Tab Bar"
  @e5 [other] "Horizontal scroll bar, 1 page"
  @e7 [scroll-area]  "Agent Device Tester"
  ...

There is a horizontal scroll bar and a tab bar alongside the vertical [scroll-area]. If our scroll gesture resolves to the wrong scrollable (the horizontal one, or the tab bar's), we would swipe repeatedly, nothing would move vertically, and the target would never appear — which matches the observed shape exactly: a clean 20s timeout rather than an overshoot or a miss.

Suggested first checks, in order:

1. Which element does our scrollUntilVisible pick as the scrollable, and does it pick the vertical [scroll-area]? Compare against the container Maestro drives.
2. Does the gesture actually move content — capture the hierarchy before/after one scroll step and diff the rects. If nothing moves, it is container selection, not the visibility predicate.
3. Only then revisit the visibility predicate and the speed→duration mapping.

Note the app screen is unremarkable — a plain ScrollView (HomeScreen.tsx:45) with a title and four buttons. If we cannot scroll this reliably, the bug is likely to affect ordinary RN apps, not an exotic layout.

Artifacts on the run (.agent-device/test-artifacts/…) include the full hierarchy capture and replay-timing.ndjson.