fix(replay): retarget identity-empty press containers to their labeled descendant (#1280)

[thymikee]

Summary

Fixes #1280 — sibling to #1269 / #1272. An Android list-row press targets a clickable container
(role="linearlayout") with no id, no label, no value — its visible title lives on a labeled
descendant (the android:id/title TextView, whose own id #1272 already demotes for being
non-unique). The container's identity is role-only and shared by every row, so replay disambiguates
purely positionally and mis-binds under reorder (measured matchCount: 12, 20/20 identity-mismatch,
per the issue). #1272's id demotion can't cover it: a press container has no id to demote.

The design

Record-time retarget to the labeled descendant, via a recording-only side channel. When a
click/press resolves to an identity-empty container, the RECORDED step (the .ad line's selector
chain and the target-v1 evidence) keys off the container's first labeled descendant — while the live
response stays container-based end-to-end.

1. Identity-empty container = resolved winner with no id surviving fix: demote non-unique ids from writer identity/selector chain (#1269) #1272's demotion (absent, or
   demoted for being non-unique), no label, and no value — all evaluated from the DEMOTED identity
   view, so a container carrying its own duplicated id does not read as identity-bearing through
   extractNodeText's raw-identifier fallback.
2. Eligible descendant = the FIRST descendant in document order (pre-order index — decision 3's
   canonical total order) with a non-empty normalized label, searched over the container's whole subtree.
3. Fail-closed guard, two conditions:
   • No competing interactive descendant: nothing in the subtree flagged hittable, and nothing the
     canonical interactive-role classification (isSemanticTouchTarget, core/interaction-targeting.ts
     — the same policy hittable-ancestor promotion uses; no parallel list) names as independently
     tappable. A row with a trailing Switch/Checkbox must NOT retarget — a tap at the
     descendant-label center vs. the container center could activate different controls.
   • Geometric containment: the selected descendant's rect center must lie INSIDE the container's
     rect — the replay tap point must be provably within the original activation region. Missing rects
     fail closed.
     When either blocks (or no labeled descendant exists), the container records exactly as today.
4. fill is excluded: its recorded chain carries editable=true constraints a label descendant
   cannot satisfy — retargeting would save an unreplayable script. Scope is click/press only.

Recording-only side channel. The runtime response — node, selectorChain, refLabel, point,
resolution disclosure, targetHittable/hint — describes the DISPATCHED container end-to-end
(preserving the runtime-selector response-identity contract). The retarget travels as an optional
recordingTarget: { node, selectorChain, refLabel } on the runtime result, consumed exclusively at
the recording boundary (interaction-touch-response.ts): the recorded action entry (the .ad
writer's result.selectorChain source) and computeTargetEvidence's node take the descendant; both
wire payloads keep container materials; the side channel is never serialized.

Scope: recording-time only. resolveSelectorChain, dispatch, and the live tap point are
untouched. Replay re-resolves the descendant and taps its center, which the guard proved lies inside
the original activation region with no competing control.

Implementation map

• src/core/press-retarget.ts — resolvePressRecordingTarget (rules 1–3). Lives in core beside
  the canonical isSemanticTouchTarget it consumes (a selectors → core import would be a layering
  back-edge).
• src/commands/interaction/runtime/resolution.ts — describeResolvedInteractionNode stays
  container-based; pressRecordingTargetOverride computes the side channel once (node + chain +
  ref-label together, so the recorded entry and its evidence can never half-retarget).
• src/daemon/handlers/interaction-touch-response.ts — the recording boundary: the recorded/history
  payload takes the descendant chain/ref-label; recordedTargetCapture prefers the side-channel node
  for evidence; responseData untouched.
• src/contracts/interaction.ts — RecordingTargetOverride, optional on ref/selector runtime results.
• Shared predicates: demoteNonUniqueLocalIdentity (extracted from fix: demote non-unique ids from writer identity/selector chain (#1269) #1272's writer, reused by the
  identity-empty check) and exported isSemanticTouchTarget.

Tests

• Unit (src/core/press-retarget.test.ts): retarget fires on the Android list-row PRESS: clickable container has no selective identity (label on child) — the identity gap #1269's id-demotion can't cover #1280 fixture shape; descendant's own
  shared id demoted per fix: demote non-unique ids from writer identity/selector chain (#1269) #1272 (chain = role+label); duplicated-container-id fixture retargets (P2a) with
  a unique-id contrast; hittable-Switch and role-typed-Checkbox guards; nested-Cell (canonical role the
  old private list missed) blocks; out-of-bounds descendant center blocks; rect-less container blocks;
  RN FlatList iOS parity; cross-invariant (chain resolves back to the exact evidence node) across
  retargeted/guard-blocked/normal cases.
• Runtime (resolution.test.ts): press response is entirely container-based with the descendant only on
  recordingTarget; fill on an identity-empty editable container does NOT retarget and its recorded
  chain (role="edittext" editable=true) still resolves uniquely — the saved script stays replayable.
• Daemon/session route (interaction-target-evidence.test.ts): record a retargeted press through
  handleInteractionCommands → the daemon response is container-based (chain, hittability, no
  side-channel leak) while the recorded entry, its target-v1 evidence, AND the written .ad line are
  descendant-based.
• Reorder e2e (session-replay-target-classification.test.ts): record a press on a row container via
  the retarget → reorder + insert a row → classifyReplayTarget rebinds the descendant by role+label
  (matchCount: 1, verified); the un-retargeted container recording refuses under the same reorder.

Docs

ADR 0012 decision 3 amendment (#1280) + Status bullet: side-channel design, click/press-only scope,
demoted-view identity-empty rule, both guard halves (canonical classification + geometry), and the
explicit recording-only statement.

Live validation — pending

Post-fix live Android record → reorder/insert → replay device evidence is pending and coordinated
separately as the shared wave-3 validation flow (the n=20 attachment on #1280 is the pre-fix failure
baseline). Per the first review round, the validation flow's recorded-artifact assertion (step-3 chain
must belong to the descendant) runs first; if the guard's hittable === true clause blocks on real
trees, the exemption decision is a follow-up with that capture as evidence.

Remarks for the maintainer

• The canonical SEMANTIC_TOUCH_ROLE_FRAGMENTS list does not name Android's edittext (the old
  private list did). The guard now blocks it only via the hittable flag or, for fill semantics,
  never reaches it (fill excluded). If an always-block for editable roles is wanted, that's a one-line
  addition to the canonical list — deliberately not made unilaterally since it also changes
  hittable-ancestor promotion.
• The click @ref native-ref fast path records no chain at all (pre-existing, fix: demote non-unique ids from writer identity/selector chain (#1269) #1272-declared gap) —
  unchanged here.

Links: #1280, #1269, #1272.

Test plan

• pnpm typecheck
• pnpm exec oxlint --deny-warnings
• pnpm format:check
• pnpm check:layering
• pnpm check:fallow --base origin/main (exit 0; single inherited duplication finding)
• pnpm check:affected --base origin/main --run (all tests pass; one run tripped only the slow-test wall-clock gate on pre-existing contention-sensitive tests while a parallel worktree ran vitest — both files re-pass within budget in isolation and are untouched by this diff)

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	1.7 MB	1.7 MB	+647 B
JS gzip	557.4 kB	557.6 kB	+224 B
npm tarball	673.1 kB	673.5 kB	+439 B
npm unpacked	2.4 MB	2.4 MB	+1.3 kB

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	27.5 ms	27.7 ms	+0.2 ms
CLI --help	57.9 ms	57.6 ms	-0.3 ms

Top changed chunks:

Chunk	Raw diff	Gzip diff
dist/src/tv-remote.js	+860 B	+274 B
dist/src/interaction.js	+188 B	+52 B
dist/src/runtime.js	-146 B	-36 B
dist/src/sdk-selectors.js	+130 B	+36 B
dist/src/selector-runtime.js	-96 B	-27 B

[thymikee]

Review of 9728b995a: the mechanism is right and honestly flagged — one blocker (the fixer's own flag 1 is real and must be fixed, not just flagged), one guard question the live validation must arbitrate, and adjudications for the other flags.

What's right: single substitution point feeding both writers (no half-retarget possible by construction); the shared demoteNonUniqueLocalIdentity extraction; cycle-safe descendant walk with document-order subtree; identity-empty test built on the #1269-demoted view; correct scoping to click/press/fill; reorder e2e present.

Blocker — flag 1 violates the design's own rule 5 (recording-time only). describeResolvedInteractionNode now computes describeNonHittableTarget(recordedNode, …) — so every retargeted press's LIVE response carries the DESCENDANT's hittability. Row-title TextViews are routinely hittable: false, which means the happy path this fix exists to serve would respond with targetHittable: false plus the #1040-style "may not be interactive" warning — a false negative disclosure about a tap that actually hit the hittable container. Required split, decided per field by what it is FOR:

• Recording-coupled fields (become the .ad step): selectorChain, evidence source (node as consumed by recordedTargetCapture), refLabel (it's the trailing label written to the .ad line) → the DESCENDANT. Correct as implemented.
• Response-semantic fields (describe what was actually dispatched): describeNonHittableTarget (targetHittable + hint) → the ORIGINAL container. Must be reverted to node.
  Document the split in the function comment, and add the test: a retargeted press response reports the CONTAINER's hittability (true) with no non-hittable hint, while its chain/evidence belong to the descendant.

Guard question — the blanket hittable === true clause may nullify the fix on real trees. isCompetingInteractive blocks on ANY hittable descendant, including the chosen labeled title itself (flag 2). For typed controls (switch/checkbox/button/…) blocking is correct — a separately-focusable control genuinely changes activation semantics. But a row title marked merely hittable is the NORMAL case on some captures, and tapping the title activates the row (it lies inside the container's touch target) — that is exactly how the corrective repair press worked in the #1235 live runs. If real Android Settings trees mark title TextViews hittable, the guard blocks on the measured repro shape and the fix never fires live. Resolution: keep the type-fragment blocking as-is; run the shared validation flow's recorded-artifact assertion (step-3 chain must belong to the descendant) FIRST — if the guard blocks on the live tree, exempt the chosen labeled descendant from the hittable === true clause (but never from the type-fragment clause) in a follow-up commit with the live capture as evidence. Do not pre-emptively weaken it without that evidence.

Other flags adjudicated:

• Flag 2 (guard blocks when the chosen descendant is interactive): correct-by-design for typed controls — see above for the hittable-only nuance. The ADR amendment should state both halves.
• Flag 3 (extractNodeText falling back to a raw undemoted node.identifier makes a container with its own non-unique id skip retargeting): real but narrow; not in the measured shape. Post it as a note on Android list-row PRESS: clickable container has no selective identity (label on child) — the identity gap #1269's id-demotion can't cover #1280 so it survives as a follow-up refinement — the identity-empty predicate should eventually consume the demoted view everywhere.
• Flag 4 (native-ref preflight fast path records no chain): pre-existing, fix: demote non-unique ids from writer identity/selector chain (#1269) #1272-declared, out of scope. Agreed.

With the flag-1 fix + its test, this is ready for the live validation flow; merge judgment after both.

[thymikee]

Blocker fixed in 686cb704b per the field-by-field split:

• Recording-coupled fields (become the .ad step) still follow the retargeted descendant: node as evidence source, selectorChain, refLabel.
• Response-semantic fields reverted to the ORIGINAL dispatched node: describeNonHittableTarget (targetHittable + hint) now describes the container actually tapped — a hittable row with a non-hittable title TextView no longer reports a false targetHittable: false on the retargeted happy path.
• The split is documented in the describeResolvedInteractionNode comment, and the ADR decision-3 amendment now states it explicitly (chain/evidence/ref-label retarget; dispatch and response hittability unchanged).

New test: runtime press #1280 retarget: recording fields follow the labeled descendant, response hittability stays on the dispatched container (src/commands/interaction/runtime/resolution.test.ts) — verified load-bearing: it fails against the pre-fix describeNonHittableTarget(recordedNode, …) line and passes with the split.

Guard (hittable === true clause) left untouched per the review — deferred to the live validation flow's recorded-artifact assertion.

Gates re-run on 686cb704b: pnpm typecheck, oxlint --deny-warnings, format:check, check:layering, check:fallow --base origin/main (exit 0, same single inherited finding), check:affected --base origin/main --run — all green (slow-test warnings within the contention band, non-failing).

[thymikee]

Re-review at 686cb704b297e63c51d23cbf96950b8558d0eb2a: not ready yet.

• P1 — the recording-only invariant is still violated. describeResolvedInteractionNode substitutes the descendant into the runtime result’s node, selectorChain, and refLabel, while point, the original target selector, resolution, and hittability describe the container actually dispatched. That makes the live/public response internally contradictory and breaks the existing runtime-selector response-identity contract. Preserve the live resolved container and carry a separate recording-only target/chain side channel. Add a daemon/session recording regression proving the response stays container-based while the written .ad chain and target-v1 evidence are descendant-based.

• P1 — fill can become unreplayable. The same retarget runs for fill; its descendant chain gets editable=true even when the labeled title descendant is not editable, so the saved fill selector cannot resolve. Scope this behavior to click/press unless fill eligibility proves an editable equivalent target, and cover the resulting script with a replay test.

• P2 — duplicate container IDs bypass retargeting. The ID is correctly demoted for non-uniqueness, but extractNodeText then reintroduces the raw identifier via its fallback, so the node is treated as identity-bearing contrary to the stated “no surviving ID” rule. Evaluate text from the demoted identity view and add the duplicated-container-ID fixture.

• P2 — the fail-closed safety guard is incomplete. Its private role fragments omit canonical interactive roles, and there is no geometry condition proving the selected descendant’s center lies inside the original activation rect. Reuse/consolidate with the canonical actionability policy and test omitted-role plus out-of-bounds descendant shapes.

All CI checks are green and GitHub currently reports mergeable, but the branch is behind current main. There is also no post-fix live Android record → reorder/insert → replay evidence yet; the attached n=20 evidence is the pre-fix failure baseline. Please provide that device evidence after the code blockers are resolved.

[thymikee]

Re-review addressed. Rebased onto current main (clean, verified file-by-file) and pushed the corrected design. Finding → fix → test:

P1 (side channel) — the runtime response is now ENTIRELY container-based. describeResolvedInteractionNode no longer substitutes anything into the response: node, selectorChain, refLabel, point, resolution disclosure, and hittability all describe the dispatched container, restoring the response-identity contract. The retarget travels as an optional recordingTarget: { node, selectorChain, refLabel } side channel on the runtime result (RecordingTargetOverride, contracts/interaction.ts), consumed exclusively at the recording boundary in interaction-touch-response.ts: the recorded action entry (the .ad writer's result.selectorChain source) takes the descendant chain/ref-label, recordedTargetCapture feeds the descendant node to computeTargetEvidence, and both wire payloads keep container materials — the side channel is never serialized.
Test: press on an identity-empty container: container-based daemon response, descendant-based recorded entry + .ad script (interaction-target-evidence.test.ts) — records a retargeted press through handleInteractionCommands, asserts the daemon response is container-based with no side-channel leak, then asserts the recorded entry, its target-v1 evidence, AND the physically written .ad line are descendant-based.
Runtime-level: runtime press #1280 retarget: the response is entirely container-based; the descendant rides only the recordingTarget side channel (resolution.test.ts).

P1 (fill) — removed from scope. Retarget is click/press only (pressRecordingTargetOverride).
Test: runtime fill #1280: fill is excluded from retargeting — the chain stays on the editable container and resolves for replay (resolution.test.ts) — a fill on an identity-empty editable container records role="edittext" editable=true, no side channel, and the chain re-resolves uniquely on the record tree (replayable script).

P2 (duplicate container ids) — identity-empty now evaluates from the DEMOTED view. The extractNodeText probe (whose raw-identifier fallback resurrected demoted ids) is gone; the check is: no id surviving demotion, no label, no value.
Tests: resolvePressRecordingTarget P2a: a container whose own duplicated id was demoted still retargets (no raw-identifier bypass) + the unique-id contrast case (src/core/press-retarget.test.ts).

P2 (guard) — canonical policy + geometry. The private fragment list is deleted; the guard is hittable === true || isSemanticTouchTarget(node) — isSemanticTouchTarget newly exported from core/interaction-targeting.ts, the same classification hittable-ancestor promotion uses (this required moving the module to src/core/press-retarget.ts; a selectors → core import is a layering back-edge). Added the geometric containment condition: the selected descendant's rect center must lie inside the container's rect, missing rects fail closed.
Tests: P2b: a nested Cell descendant (canonical role the old fragment list missed) blocks retargeting, P2b: a labeled descendant whose rect center lies OUTSIDE the container rect blocks retargeting, P2b: a rect-less container blocks retargeting (geometry fails closed).

ADR decision-3 amendment + Status bullet rewritten: side-channel design, click/press-only scope, demoted-view rule, both guard halves, recording-only statement. PR body updated to match, including the note that post-fix live record → reorder → replay device evidence is pending and coordinated separately (shared wave-3 validation flow); the hittable === true exemption question stays deferred to that flow per the first review round.

One remark: the canonical SEMANTIC_TOUCH_ROLE_FRAGMENTS doesn't name Android's edittext (the deleted private list did) — such a descendant now blocks only via the hittable flag. Adding it to the canonical list would also change hittable-ancestor promotion, so I didn't do it unilaterally.

Gates on the rebased tree: typecheck, oxlint --deny-warnings, format:check, check:layering, check:fallow --base origin/main (exit 0; single inherited duplication finding) all green; check:affected --base origin/main --run: all tests pass; the run tripped only the slow-test WALL-CLOCK gate on the pre-existing contention-sensitive session-replay-vars/daemon-entrypoint tests while a parallel worktree ran vitest concurrently — both files re-pass within budget in isolation and are untouched by this diff. The new daemon regression test itself was moved onto the iOS runtime path (direct-iOS stays recording-gated) so the unit lane spends no real wall-clock on Android adb dialog probes (1.9s → 8ms).

PR stays draft pending the live device evidence.

[thymikee]

Exact-head re-review (c45f2d61f): code review is clean. The response remains entirely container-based; the descendant travels only through the recording-only side channel, consumed at the single recording boundary so recorded selector/evidence change together without leaking onto the wire. Fill is excluded and its editable container chain is replayable. Duplicate IDs are evaluated after demotion, and the guard now reuses canonical semantic actionability plus fails-closed geometry checks. The daemon-route regression exercises the real handler, verifies wire/history separation, and physically pins the .ad line; the reorder test covers rebinding.

All current CI is green and the PR is mergeable. Residual pre-merge evidence: the post-fix live Android record → reorder/insert → replay run is still pending; attach it before merge, particularly to validate real-device hittability/guard behavior. This is not a remaining code-review finding.

[thymikee]

Live device evidence, part 1 of 2 — the hittability/guard question is answered: the retarget fires on a real Android Settings tree and the recorded artifact is exactly the designed shape.

Setup: combined tree cae03de8d (this PR's head c45f2d61f + #1285 1089e7c06 + #1288 5046190f7, all on main 1a1ef7c41), pnpm build && pnpm build:android, emulator-5556, helper path verified (backend: "android-helper", no fallback). Archive: ~/.agent-device-bench/wave3/validation/.

Recorded press step from the live flow (scripts/flowA-settings.ad) — the identity-empty linearlayout row container was retargeted to its labeled descendant, with verified evidence:

# agent-device:target-v1 {"role":"textview","label":"Connected devices","ancestry":[{"role":"linearlayout"},{"role":"recyclerview"},...],"sibling":0,"viewportOrder":0,"scrollRegion":{"role":"scrollview","id":"com.android.settings:id/main_content_scrollable_container"},...,"verification":"verified"}
press "role=\"textview\" label=\"Connected devices\" || label=\"Connected devices\" || value=\"Connected devices\""

No android:id/title id, no bare container role — and the LIVE press response carried selectorChain: ["role=\"linearlayout\""] (container-based wire), confirming the recording-only side channel end-to-end on device. The guard did not over-block on real hittability data (the parked round-1 hittable === true question closes as no-change-needed). Sanity replay of the recorded flow: 9/9 steps clean, 0 healed, 34.2s.

Part 2 (pending a low-contention window): the n=20 replay protocol and the deterministic Battery record→insert→replay leg. A first n=20 attempt was deliberately stopped at 2/20 when this shared machine's load hit 101–166 (unrelated concurrent sessions) and produced 30s adb capture-timeouts — those two runs are archived as environment artifacts, not counted. The fresh n=20 + insert leg run automatically when the machine quiets and will be posted here.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-16 17:18 UTC