Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion src/cli/parser/__tests__/cli-help-topics.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -238,8 +238,10 @@ test('usageForCommand resolves workflow help topic', async () => {
assert.match(help, /To hide the keyboard, use keyboard dismiss/);
assert.match(
help,
/press a visible app control such as Done only when that is the intended fallback/,
/On iOS, if it returns UNSUPPORTED_OPERATION, there is no generic app-agnostic blur fallback/,
);
assert.match(help, /On Android, keyboard dismiss first avoids navigation/);
assert.match(help, /use back only when normal back behavior is acceptable/);
assert.match(help, /UNSUPPORTED_OPERATION/);
assert.match(help, /Stateful commands within one session must run serially/);
assert.match(
Expand Down
4 changes: 3 additions & 1 deletion src/cli/parser/cli-help.ts
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,9 @@ Text entry:
agent-device type "Handle with care" --delay-ms 80
Empty replacement is not a supported clear-field command: do not plan fill <target> "" or fill <target> ''. Prefer a visible clear/reset control; if the app exposes none, report the tool gap instead of inventing a clear command.
Debounced field with no result selector: agent-device wait 1000. Keyboard read-only: keyboard status/get. The on-screen keyboard usually does not block agent-device interactions; press the next target directly instead of dismissing. If that press fails or reports no visible effect, scroll the target into view or use keyboard enter when submission is wanted.
Only dismiss the keyboard when hiding it is the actual goal. To hide the keyboard, use keyboard dismiss. It taps safe controls like Done when available and verifies the keyboard closed. If it reports UNSUPPORTED_OPERATION, press a visible app control such as Done only when that is the intended fallback, or press the next target directly instead of retrying dismiss.
Only dismiss the keyboard when hiding it is the actual goal. To hide the keyboard, use keyboard dismiss. It taps safe controls like Done when available and verifies the keyboard closed.
On iOS, if it returns UNSUPPORTED_OPERATION, there is no generic app-agnostic blur fallback: do not assume a static text or heading is safe to press, because it can belong to a tappable parent. Use an app-provided dismiss control only when its action is explicitly intended; otherwise report that keyboard dismissal is unavailable.
On Android, keyboard dismiss first avoids navigation. If it returns UNSUPPORTED_OPERATION because the current IME needs back navigation, use back only when normal back behavior is acceptable; otherwise report that keyboard dismissal is unavailable.
Use plain fill/type first for ordinary login and form fields. If an iOS debounced or search-as-you-type field actually drops characters, or must receive incremental updates, retry with --delay-ms before trying clipboard paste; --delay-ms intentionally paces character entry.
iOS Allow Paste prompt cannot be exercised under XCUITest. To test paste-driven app behavior, prefill first with agent-device clipboard write "some text"; test the system prompt manually.
Android Gboard handwriting/stylus UI can capture text in an IME-owned input instead of the app field. If fill reports that input was captured by the keyboard/IME, use the diagnostic targetInput/actualInput details, inspect keyboard status/get if needed, and switch or disable handwriting outside the command plan before retrying. Do not keep retrying fill/type against the same field while the IME owns focus.
Expand Down
79 changes: 72 additions & 7 deletions test/skillgym/suites/agent-device-smoke-suite.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ Use only this prompt plus local CLI help as private reference.
Do not execute live app/device commands while planning; only local CLI help commands are allowed before final output.
For local CLI help in this repo, use node bin/agent-device.mjs help or --help; final commands still use agent-device.
If the app contract names an expected id, selector, or visible text, include that exact target in a final verification command instead of stopping at the action that reaches or reveals it.
`.trim();

const DEFAULT_FINAL_OUTPUT_INSTRUCTIONS = `
Final output: only commands, one per line. Use agent-device for app/device automation; shell setup commands are allowed only when this prompt explicitly requires them. Any prose or Markdown fails.
Every final output line must start with agent-device.
Do not combine final commands with shell operators such as &&, ||, pipes, or semicolons.
Expand All @@ -62,9 +65,15 @@ function escapeRegExp(value: string) {
return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}

function buildPrompt(options: { contract: string[]; task: string }) {
function buildPrompt(options: {
contract: string[];
task: string;
finalOutputInstructions?: string;
}) {
const contractLines = options.contract.map((line) => `- ${line}`).join('\n');
return `${BASE_INSTRUCTIONS}\n\nApp contract:\n${contractLines}\n\nTask:\n${options.task}`;
const finalOutputInstructions =
options.finalOutputInstructions ?? DEFAULT_FINAL_OUTPUT_INSTRUCTIONS;
return `${BASE_INSTRUCTIONS}\n${finalOutputInstructions}\n\nApp contract:\n${contractLines}\n\nTask:\n${options.task}`;
}

function assertAgentDeviceEvidence(report: SessionReport) {
Expand Down Expand Up @@ -451,11 +460,16 @@ function makeCase(options: {
forbiddenOutputs?: OutputMatcher[];
strictFinalOutput?: boolean;
allowOnlyLocalCliHelpCommands?: boolean;
finalOutputInstructions?: string;
}): Case {
return {
id: options.id,
tags: options.tags,
prompt: buildPrompt({ contract: options.contract, task: options.task }),
prompt: buildPrompt({
contract: options.contract,
task: options.task,
finalOutputInstructions: options.finalOutputInstructions,
}),
assert(report, ctx) {
assertAgentDeviceEvidence(report);
assertNoProjectSourceReads(report);
Expand Down Expand Up @@ -692,11 +706,62 @@ const FIXTURE_SMOKE_CASES: Case[] = [
'Current screen: Checkout form tab',
'testID=field-name',
'keyboard dismiss already returned UNSUPPORTED_OPERATION',
'visible app keyboard close control: Done',
'Visible text "Order summary" belongs to a tappable accordion',
'testID=submit-order is partly occluded, and pressing it would submit the form',
'No app-provided keyboard close control is available',
],
task: 'Decide whether there is a safe app-agnostic fallback that can hide the keyboard.',
finalOutputInstructions:
'Final output: one concise prose sentence stating the safe outcome. Do not output commands or Markdown.',
outputs: [/(?:keyboard dismissal is unavailable|cannot safely dismiss the keyboard)/i],
forbiddenOutputs: [
plannedCommand('keyboard dismiss'),
plannedCommand('back'),
plannedCommandAlternatives(['press', 'click']),
/agent-device/i,
/(?:press|click)[^\n]*(?:Order summary|submit-order)/i,
],
}),
makeCase({
id: 'form-keyboard-dismiss-ios-app-control-fallback',
contract: [
'App name: Agent Device Tester',
'Platform: iOS simulator',
'Current screen: Checkout form tab with the keyboard visible',
'keyboard dismiss already returned UNSUPPORTED_OPERATION',
'The focused field accessory exposes a Done control',
'Visible text "Order summary" belongs to a tappable accordion',
'testID=submit-order is occluded while the keyboard is visible',
],
task: 'Plan the safe fallback commands to hide the keyboard and prove that the previously occluded Submit order target became visible.',
outputs: [
plannedCommandAlternatives(['press', 'click']),
plannedCommand('is visible'),
/(?:press|click)[^\n]*Done[\s\S]*is visible[^\n]*submit-order/i,
],
forbiddenOutputs: [
plannedCommand('keyboard dismiss'),
plannedCommand('back'),
/(?:press|click)[^\n]*Order summary/i,
],
}),
makeCase({
id: 'android-keyboard-dismiss-back-when-allowed',
contract: [
'App name: Agent Device Tester',
'Platform: Android',
'Current screen: Checkout form tab with the keyboard visible',
'testID=field-name',
'keyboard dismiss already returned UNSUPPORTED_OPERATION because the current IME requires navigation to close',
'Moving one level toward the previous UI state is acceptable for this task',
],
task: 'Plan the safe fallback commands to hide the keyboard and verify the Full name field remains visible.',
outputs: [
plannedCommand('back'),
plannedCommand('is visible'),
/back[\s\S]*is visible[\s\S]*field-name/i,
],
task: 'Assume Agent Device Tester is on the Checkout form tab. Plan the fallback commands to focus the Full name field, close the iOS keyboard through the visible app control, and verify the field remains visible.',
outputs: [/field-name/i, /Done/i, plannedCommandAlternatives(['press', 'click'])],
forbiddenOutputs: [plannedCommand('keyboard dismiss'), plannedCommand('back')],
forbiddenOutputs: [plannedCommand('keyboard dismiss'), RAW_COORDINATE_TARGET],
}),
makeCase({
id: 'form-keyboard-dismiss-ios-done-control',
Expand Down
Loading