Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
165 changes: 158 additions & 7 deletions docs/adr/0012-interactive-replay.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions scripts/integration-progress-model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,7 @@ function summarizeProviderScenarioFlagCoverage(files) {
['restart', 'logs clear --restart workflow'],
['networkInclude', 'network dump include modes', ['include']],
['noRecord', 'action recording suppression'],
['record', 'repair-segment observation-only recording opt-in (ADR 0012)'],
['replayUpdate', 'retired --update no-op replays without rewriting (ADR 0012)', ['update']],
['replayEnv', 'replay/test variable injection', ['env']],
['replayFrom', 'replay resume skips completed steps (ADR 0012)', ['resumeFrom']],
Expand Down
2 changes: 1 addition & 1 deletion src/cli/parser/cli-help.ts
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,7 @@ Validation and evidence:
agent-device screenshot
agent-device press 124 817
agent-device snapshot -i
Startup/CPU/memory/frame first pass: perf metrics --json (bare perf and metrics are aliases). Focused frame/jank health: perf frames --json. Memory-only sample: perf memory sample --json returns compact JSON with bounded top offenders. Heap/memgraph artifact escalation: perf memory snapshot --out heap.artifact; use --kind android-hprof on Android or --kind memgraph on supported Apple simulator/macOS app sessions. Android native profiling: perf cpu profile start|stop|report --kind simpleperf --out <path>; Android native traces: perf trace start|stop --kind perfetto --out <path>. Artifact collectors return compact state/path/size metadata only; raw heap/profile/trace files stay on disk. Treat native perf output as the agent evidence: for example, a Perfetto stop can return state=stopped, outPath=/tmp/app.perfetto-trace, sizeBytes=5392410, and method=adb-shell-perfetto while the 5.3 MB raw trace stays in the artifact. This is better than raw dumps for agents because it is stable, bounded, and keeps large artifacts out of context. heapprofd is deferred until Perfetto plumbing is available. Replay divergence and resume: a failing replay/test step returns REPLAY_DIVERGENCE with a bounded report (screen digest, ranked selector suggestions, resume). Repair app state, then resume with replay --from <n> --plan-digest <sha256> (both from the report's resume field) to continue from the failed step without re-running earlier ones; resume never re-executes skipped steps, so app state is the caller's responsibility, and it is rejected with INVALID_ARGS when the plan digest is stale, --from is out of range, or the skipped range/target touches runtime control flow. The plan digest binds the script, its includes, the effective --platform/--target, and per-action runtime/identity. Native .ad interpolation is late-bound after planning, so changing only its values keeps the digest; Maestro environment substitution occurs during compatibility parsing and can change action inputs, includes, or control expansion, so it can change the digest. --from is replay-only; test rejects it. --update/-u no longer rewrites the script (ADR 0012) — it is a no-op kept for compatibility; every divergence already carries the same ranked suggestions. Agent-supervised repair (heal-by-doing): arm replay <file>.ad --save-script[=<out>] before step 1 (armed once; --from continuations do not need it again). Every divergence carries a repairHint: record-and-heal means press the correct control via a blessed @ref from the divergence's screen.refs, recorded (no --no-record), then continue with replay --from <n+1> --plan-digest <sha256>; state-repair means the script is correct but app state is not, so fix state with --no-record actions, then replay --from <n> --plan-digest <sha256> to re-run the unchanged step; caution means something already matches the recorded selector, so a blind re-press may repeat the mistake; manual means no safe automated repair could be proven. While armed, read-only inspection used to locate the repair target (snapshot -i, get attrs, find, is) is recorded too — pass --no-record on those commands, or they land in the healed script alongside the corrective action. Ending the repair with close --save-script[=<out>] writes only the steps recorded since the arming replay as <out>, defaulting to the <file> sibling <stem>.healed.ad — review its diff before promoting it over the original.
Startup/CPU/memory/frame first pass: perf metrics --json (bare perf and metrics are aliases). Focused frame/jank health: perf frames --json. Memory-only sample: perf memory sample --json returns compact JSON with bounded top offenders. Heap/memgraph artifact escalation: perf memory snapshot --out heap.artifact; use --kind android-hprof on Android or --kind memgraph on supported Apple simulator/macOS app sessions. Android native profiling: perf cpu profile start|stop|report --kind simpleperf --out <path>; Android native traces: perf trace start|stop --kind perfetto --out <path>. Artifact collectors return compact state/path/size metadata only; raw heap/profile/trace files stay on disk. Treat native perf output as the agent evidence: for example, a Perfetto stop can return state=stopped, outPath=/tmp/app.perfetto-trace, sizeBytes=5392410, and method=adb-shell-perfetto while the 5.3 MB raw trace stays in the artifact. This is better than raw dumps for agents because it is stable, bounded, and keeps large artifacts out of context. heapprofd is deferred until Perfetto plumbing is available. Replay divergence and resume: a failing replay/test step returns REPLAY_DIVERGENCE with a bounded report (screen digest, ranked selector suggestions, resume). Repair app state, then resume with replay --from <n> --plan-digest <sha256> (both from the report's resume field) to continue from the failed step without re-running earlier ones; resume never re-executes skipped steps, so app state is the caller's responsibility, and it is rejected with INVALID_ARGS when the plan digest is stale, --from is out of range, or the skipped range/target touches runtime control flow. The plan digest binds the script, its includes, the effective --platform/--target, and per-action runtime/identity. Native .ad interpolation is late-bound after planning, so changing only its values keeps the digest; Maestro environment substitution occurs during compatibility parsing and can change action inputs, includes, or control expansion, so it can change the digest. --from is replay-only; test rejects it. --update/-u no longer rewrites the script (ADR 0012) — it is a no-op kept for compatibility; every divergence already carries the same ranked suggestions. Agent-supervised repair (heal-by-doing): arm replay <file>.ad --save-script[=<out>] before step 1 (armed once; --from continuations do not need it again). Every divergence carries a repairHint: record-and-heal means press the correct control via a blessed @ref from the divergence's screen.refs, recorded (no --no-record), then continue with replay --from <n+1> --plan-digest <sha256>; state-repair means the script is correct but app state is not, so fix state with --no-record actions, then replay --from <n> --plan-digest <sha256> to re-run the unchanged step; caution means something already matches the recorded selector, so a blind re-press may repeat the mistake; manual means no safe automated repair could be proven. While armed, read-only inspection YOU run to locate the repair target (snapshot -i, get attrs, find, is) is excluded from the healed script by default — no --no-record needed; the script's own authored get/is/find steps are unaffected and stay in the heal. If the step you are repairing is itself a read, pass --record on that one command so it lands in the heal (--record is accepted only on snapshot/get/is and a read-only find; it is mutually exclusive with --no-record). Ending the repair with close --save-script[=<out>] writes only the steps recorded since the arming replay as <out>, defaulting to the <file> sibling <stem>.healed.ad — review its diff before promoting it over the original.
Recording: record start/stop. The default scope is app and expects an active session created by open <app>; this keeps app proof videos tied to the intended app session. Use record start --scope device/system to explicitly request whole-screen capture where the selected backend supports it, such as recordings that intentionally span multiple apps, home screen, settings, or app transitions. Use --max-size to cap the longest edge and --quality medium|high to choose output quality across Android and Apple targets. By default, stop burns touch overlays into the video; use record start --hide-touches for the fastest raw recording. Android record start publishes a durable device manifest. Android adb screenrecord has a 180s platform limit, so longer Android recordings are returned as multiple MP4 chunks while the daemon stays alive; after daemon restart, record stop recovers only manifest-owned chunks and warns when gesture overlays are unavailable. For gesture-heavy iOS simulator proof videos, prefer --hide-touches because overlay timing depends on a stable runner session while gestures are executing. Tracing: trace start ./trace.log, trace stop ./trace.log. Paths are positional.
Stable known flow: batch ./steps.json, not workflow batch.
Inline batch JSON example:
Expand Down
32 changes: 28 additions & 4 deletions src/client/client-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -456,6 +456,13 @@ export type CaptureSnapshotOptions = AgentDeviceRequestOverrides &
raw?: boolean;
forceFull?: boolean;
timeoutMs?: number;
/**
* #1271 stage 2 (ADR 0012 amendment): `snapshot` is observation-only and
* excluded from a repair-armed heal by default; `record` forces it
* through. Mutually exclusive with `noRecord`.
*/
noRecord?: boolean;
record?: boolean;
};

export type CaptureSnapshotResult = {
Expand Down Expand Up @@ -818,21 +825,35 @@ export type RotateGestureOptions = DeviceCommandBaseOptions & {

export type TransformGestureOptions = DeviceCommandBaseOptions & TransformGestureParams;

/**
* #1271 stage 2 (ADR 0012 amendment): `get`/`is`/`find` are observation-only
* and excluded from a repair-armed heal by default. `record` forces this
* action through (the corrective-read case); `noRecord` continues to opt the
* action out entirely. Mutually exclusive.
*/
type RecordControlOptions = {
noRecord?: boolean;
record?: boolean;
};

export type GetOptions = DeviceCommandBaseOptions &
SelectorSnapshotCommandOptions &
ElementTarget & {
ElementTarget &
RecordControlOptions & {
format: 'text' | 'attrs';
};

type IsTextPredicateOptions = DeviceCommandBaseOptions &
SelectorSnapshotCommandOptions & {
SelectorSnapshotCommandOptions &
RecordControlOptions & {
predicate: 'text';
selector: string;
value: string;
};

type IsStatePredicateOptions = DeviceCommandBaseOptions &
SelectorSnapshotCommandOptions & {
SelectorSnapshotCommandOptions &
RecordControlOptions & {
predicate: 'visible' | 'hidden' | 'exists' | 'editable' | 'selected' | 'focused';
selector: string;
value?: never;
Expand All @@ -841,7 +862,8 @@ type IsStatePredicateOptions = DeviceCommandBaseOptions &
export type IsOptions = IsTextPredicateOptions | IsStatePredicateOptions;

type FindBaseOptions = DeviceCommandBaseOptions &
FindSnapshotCommandOptions & {
FindSnapshotCommandOptions &
RecordControlOptions & {
locator?: FindLocator;
query: string;
first?: boolean;
Expand Down Expand Up @@ -1098,6 +1120,8 @@ export type InternalRequestOptions = AgentDeviceClientConfig &
deviceHub?: boolean;
testIme?: boolean;
noRecord?: boolean;
/** #1271 stage 2: force-record this action; mutually exclusive with `noRecord`. */
record?: boolean;
backMode?: BackMode;
metroHost?: string;
metroPort?: number;
Expand Down
16 changes: 13 additions & 3 deletions src/commands/capture/snapshot.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ import { defineExecutableCommand } from '../command-contract.ts';
import {
commonInputFromFlags,
direct,
recordControlInputFromFlags,
noRecordInputFromFlags,
observationRecordInputFromFlags,
} from '../cli-grammar/common.ts';
import type { CliReader, DaemonWriter } from '../cli-grammar/types.ts';
import { defineCommandFacet } from '../family/types.ts';
Expand All @@ -26,6 +27,14 @@ const snapshotCommandMetadata = defineFieldCommandMetadata(
raw: booleanField(),
forceFull: booleanField(),
timeoutMs: integerField('Maximum wall-clock time for the snapshot command.'),
// #1271 stage 2: `snapshot` is observation-only, so a repair-armed heal
// excludes an out-of-band one by default (ADR 0012 amendment). Exposed
// here so the Node SDK's typed options and the MCP tool schema can set
// both flags, mirroring `--no-record`/`--record` on the CLI.
noRecord: booleanField('Do not record this action.'),
record: booleanField(
'Force-record this out-of-band observation into a repair-armed heal (mutually exclusive with noRecord). Authored replay steps are recorded automatically and never need this.',
),
},
);

Expand All @@ -40,12 +49,13 @@ const snapshotCliSchema = {
helpDescription:
'Capture accessibility tree or diff against the previous session baseline. For iOS raw-coordinate fallback after a no-op ref press, inspect rects with snapshot -i --json, press the rect center, then verify with diff snapshot -i or snapshot --diff.',
summary: 'Capture accessibility tree or diff against the previous session baseline',
allowedFlags: ['snapshotDiff', ...SNAPSHOT_FLAGS, 'snapshotForceFull', 'timeoutMs'],
allowedFlags: ['snapshotDiff', ...SNAPSHOT_FLAGS, 'snapshotForceFull', 'timeoutMs', 'record'],
} as const;

export const snapshotCliReader: CliReader = (_positionals, flags) => ({
...commonInputFromFlags(flags),
...recordControlInputFromFlags(flags),
...noRecordInputFromFlags(flags),
...observationRecordInputFromFlags(flags),
interactiveOnly: flags.snapshotInteractiveOnly,
depth: flags.snapshotDepth,
scope: flags.snapshotScope,
Expand Down
4 changes: 2 additions & 2 deletions src/commands/capture/wait.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import { defineExecutableCommand } from '../command-contract.ts';
import {
direct,
optionalNumber,
recordControlInputFromFlags,
noRecordInputFromFlags,
selectionOptionsFromFlags,
selectorSnapshotOptionsFromFlags,
} from '../cli-grammar/common.ts';
Expand Down Expand Up @@ -93,7 +93,7 @@ function readWaitOptionsFromPositionals(
const base = {
...selectionOptionsFromFlags(flags),
...selectorSnapshotOptionsFromFlags(flags),
...recordControlInputFromFlags(flags),
...noRecordInputFromFlags(flags),
};
if (parsed.kind === 'sleep') return { ...base, durationMs: parsed.durationMs };
if (parsed.kind === 'text') {
Expand Down
25 changes: 24 additions & 1 deletion src/commands/cli-grammar/common.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,12 +70,35 @@ export function commonInputFromFlags(flags: CliFlags): Record<string, unknown> {
// it: command-flags.ts maps options.noRecord onto the daemon request, and
// recordActionEntry reads it from there. A reader that never names it accepts
// the flag and silently drops it.
export function recordControlInputFromFlags(flags: CliFlags): Record<string, unknown> {
//
// #1271 stage 2: `--record` deliberately does NOT ride along here. It applies
// only to observation-only commands (snapshot/get/is/find — the ones the
// repair-segment default exclusion can drop), so it gets its own helper below
// rather than an `allowRecord` policy argument on this one. The capability is
// the helper's NAME: a mutating reader physically cannot forward `--record`
// without spreading a helper whose name says it is for observations, whereas a
// boolean policy arg would let a future mutating reader opt in by flipping a
// literal with no schema change — the fail-open this split exists to prevent.
export function noRecordInputFromFlags(flags: CliFlags): Record<string, unknown> {
return compactRecord({
noRecord: flags.noRecord,
});
}

/**
* #1271 stage 2 (ADR 0012 decision 6 amendment): the `--record` opt-in that
* forces an observation-only action into a repair-armed heal. Spread ONLY by
* readers whose command can be excluded by default — `snapshot`, `get`, `is`,
* and `find`. Every one of those readers must ALSO spread
* `noRecordInputFromFlags`, which stays universal (`--no-record` applies to
* every recordable command, mutations included).
*/
export function observationRecordInputFromFlags(flags: CliFlags): Record<string, unknown> {
return compactRecord({
record: flags.record,
});
}

export function selectionOptionsFromFlags(flags: CliFlags): SelectionOptions {
return {
platform: flags.platform,
Expand Down
8 changes: 8 additions & 0 deletions src/commands/cli-grammar/flag-definitions-action.ts
Original file line number Diff line number Diff line change
Expand Up @@ -294,4 +294,12 @@ export const ACTION_FLAG_DEFINITIONS: readonly FlagDefinition[] = [
usageLabel: '--no-record',
usageDescription: 'Do not record this action',
},
{
key: 'record',
names: ['--record'],
type: 'boolean',
usageLabel: '--record',
usageDescription:
'Force-record this action even though its command is observation-only and would otherwise be excluded from a repair-armed heal by default (mutually exclusive with --no-record)',
},
];
6 changes: 6 additions & 0 deletions src/commands/cli-grammar/flag-groups.ts
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,12 @@ export const COMMON_COMMAND_SUPPORTED_FLAG_KEYS = flagKeys(
'iosXctestEnvDir',
'androidDeviceAllowlist',
'session',
// `--no-record` is genuinely common: it applies to every recordable command,
// mutations included. `--record` is NOT — it only means anything for the
// observation-only commands the repair-segment exclusion can drop
// (snapshot/get/is/find), so it is scoped per-command via each schema's
// `allowedFlags` instead of being accepted everywhere and silently ignored
// (#1271 stage 2).
'noRecord',
);

Expand Down
1 change: 1 addition & 0 deletions src/commands/command-flags.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ function buildFlags(options: InternalRequestOptions): CommandFlags {
deviceHub: options.deviceHub,
testIme: options.testIme,
noRecord: options.noRecord,
record: options.record,
backMode: options.backMode,
metroHost: options.metroHost,
metroPort: options.metroPort,
Expand Down
6 changes: 3 additions & 3 deletions src/commands/interaction/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,20 +60,20 @@ const interactionCliSchemas = {
get: {
usageOverride: 'get text|attrs <@ref|selector>',
positionalArgs: ['subcommand', 'target'],
allowedFlags: [...SELECTOR_SNAPSHOT_FLAGS],
allowedFlags: [...SELECTOR_SNAPSHOT_FLAGS, 'record'],
},
find: {
usageOverride: 'find <locator|text> <action> [value] [--first|--last]',
helpDescription: 'Find by text/label/value/role/id and run action',
summary: 'Find an element and act',
positionalArgs: ['query', 'action', 'value?'],
allowsExtraPositionals: true,
allowedFlags: ['snapshotDepth', 'snapshotRaw', 'findFirst', 'findLast'],
allowedFlags: ['snapshotDepth', 'snapshotRaw', 'findFirst', 'findLast', 'record'],
},
is: {
positionalArgs: ['predicate', 'selector', 'value?'],
allowsExtraPositionals: true,
allowedFlags: [...SELECTOR_SNAPSHOT_FLAGS],
allowedFlags: [...SELECTOR_SNAPSHOT_FLAGS, 'record'],
},
click: {
usageOverride: 'click <x y|@ref|selector>',
Expand Down
Loading
Loading