From 3d48c8b128054414c528e127778ec9d56cefea4d Mon Sep 17 00:00:00 2001 From: Mike Pitre <12040919+mikepitre@users.noreply.github.com> Date: Wed, 7 Oct 2026 20:30:57 -0400 Subject: [PATCH 1/7] chore(repo): add the Expo end-to-end test package, its entry point, and repo wiring Co-Authored-By: Claude Opus 5.5 --- .changeset/expo-verify-skill.md | 2 + .claude/skills/README.md | 19 +- .cursor/skills/verify-clerk-expo | 1 + .prettierignore | 14 + eslint.config.mjs | 1 + integration/expo-native/.gitignore | 3 + integration/expo-native/README.md | 69 + .../expo-native/bin/control-clerk-expo | 7 + integration/expo-native/e2e.config.ts | 5 + integration/expo-native/package-lock.json | 1783 +++++++++++++++++ integration/expo-native/package.json | 20 + integration/expo-native/tsconfig.json | 16 + 12 files changed, 1933 insertions(+), 7 deletions(-) create mode 100644 .changeset/expo-verify-skill.md create mode 120000 .cursor/skills/verify-clerk-expo create mode 100644 integration/expo-native/.gitignore create mode 100644 integration/expo-native/README.md create mode 100755 integration/expo-native/bin/control-clerk-expo create mode 100644 integration/expo-native/e2e.config.ts create mode 100644 integration/expo-native/package-lock.json create mode 100644 integration/expo-native/package.json create mode 100644 integration/expo-native/tsconfig.json diff --git a/.changeset/expo-verify-skill.md b/.changeset/expo-verify-skill.md new file mode 100644 index 00000000000..a845151cc84 --- /dev/null +++ b/.changeset/expo-verify-skill.md @@ -0,0 +1,2 @@ +--- +--- diff --git a/.claude/skills/README.md b/.claude/skills/README.md index 0031ac9bc9c..97f3bd4fdce 100644 --- a/.claude/skills/README.md +++ b/.claude/skills/README.md @@ -23,9 +23,13 @@ Edits to a `SKILL.md` take effect immediately, including in already-running sess ## Scope -Skills are Claude Code specific. Cursor does not read this directory; it uses `.cursor/rules/` and -`AGENTS.md`. When a repo rule changes, update `AGENTS.md` first, then mirror the change here and in -`.cursor/rules/` where relevant. +Skills here are Claude Code specific, except `verify-clerk-expo`, the agent-neutral skill that drives +`@clerk/expo` on a simulator or emulator. Its `SKILL.md` and references live here, and the tests and +the CLI it drives are the package `integration/expo-native/`. `.cursor/skills/verify-clerk-expo` is a +symlink to this directory so Cursor reads the same skill. Edit it here, not through the +symlink. For the other skills, Cursor uses `.cursor/rules/` and `AGENTS.md`. When a repo rule +changes, update `AGENTS.md` first, then mirror the change here and in `.cursor/rules/` where +relevant. ## Maintaining a skill @@ -41,7 +45,8 @@ Skills are Claude Code specific. Cursor does not read this directory; it uses `. ## Skills in this repo -| Skill | Use it for | -| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -| `clerk-monorepo` | Day-to-day work in the monorepo: setup, build/test loops, the package map, changesets, commits, PRs, breaking-change checks. | -| `mosaic` | Mosaic flow UI: authoring machines, controllers, and views, and migrating a legacy component into the split (with parity verification). | +| Skill | Use it for | +| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| `clerk-monorepo` | Day-to-day work in the monorepo: setup, build/test loops, the package map, changesets, commits, PRs, breaking-change checks. | +| `mosaic` | Mosaic flow UI: authoring machines, controllers, and views, and migrating a legacy component into the split (with parity verification). | +| `verify-clerk-expo` | Proving a `@clerk/expo` change on an iOS simulator or Android emulator, with video and screenshots as evidence. | diff --git a/.cursor/skills/verify-clerk-expo b/.cursor/skills/verify-clerk-expo new file mode 120000 index 00000000000..bb470267b68 --- /dev/null +++ b/.cursor/skills/verify-clerk-expo @@ -0,0 +1 @@ +../../.claude/skills/verify-clerk-expo \ No newline at end of file diff --git a/.prettierignore b/.prettierignore index fc97a58a963..d3d75f0ac22 100644 --- a/.prettierignore +++ b/.prettierignore @@ -29,3 +29,17 @@ CLAUDE.md # Written by `eslint --suppress-rule` / `--prune-suppressions`; ESLint owns the formatting. /**/eslint-suppressions.json + +# Copied byte for byte from another repo; do not reformat. +integration/expo-native/src/core/ +integration/expo-native/src/platform/android/ +integration/expo-native/src/platform/ios/ +integration/expo-native/specs/fixtures.ts +integration/expo-native/specs/support/ +integration/expo-native/testing/ +integration/expo-native/e2e.config.ts +integration/expo-native/test/*.ts +!integration/expo-native/test/freshness.test.ts +!integration/expo-native/test/host.test.ts +integration/expo-native/.verify/ +integration/expo-native/specs/explored/ diff --git a/eslint.config.mjs b/eslint.config.mjs index c0ae474108c..69da7c0bcbd 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -326,6 +326,7 @@ export default tseslint.config([ '**/build/*', '**/coverage/*', '**/dist/*', + '**/integration/expo-native/**/*', '**/integration/templates/**/*', '**/node_modules/**', '*.snap', diff --git a/integration/expo-native/.gitignore b/integration/expo-native/.gitignore new file mode 100644 index 00000000000..603ba938f6c --- /dev/null +++ b/integration/expo-native/.gitignore @@ -0,0 +1,3 @@ +.e2e/ +.verify/ +specs/explored/ diff --git a/integration/expo-native/README.md b/integration/expo-native/README.md new file mode 100644 index 00000000000..f8dd66d3aca --- /dev/null +++ b/integration/expo-native/README.md @@ -0,0 +1,69 @@ +# @clerk/expo end-to-end tests + +The tests in `specs/` are an ordinary [e2e](https://github.com/tester-army/e2e) project. `e2e.config.ts` and the files under `specs/` run with `npx e2e run` when the environment names three things: a device, a build of the `expo-native` test app, and a Clerk development instance. + +The CLI in `bin/` and `src/` sits on top of that project. It makes those three things for a run, and it adds what a pull request needs. The tests never import the CLI. `test/seam.test.ts` fails when a file under `specs/` or `e2e.config.ts` names a path outside `specs/`. + +To prove a change, use the CLI. `.claude/skills/verify-clerk-expo/SKILL.md` has the steps. Run a test by hand when you want e2e alone, for example to work on a test with e2e's own flags. + +## Run a test by hand + +You need Node 24.8 or newer on 24, Xcode and a booted iOS simulator, or the Android SDK and a running emulator, and the publishable key and secret key of a Clerk development instance. + +```sh +cd integration/expo-native +npm ci +export CLERK_E2E_PLATFORM=ios +xcrun simctl list devices booted +export CLERK_E2E_DEVICE= +export CLERK_E2E_APP_PATH= +export CLERK_E2E_DEV_SERVER=http://localhost:8081 +export CLERK_PUBLISHABLE_KEY=pk_test_... +export CLERK_SECRET_KEY=sk_test_... +npx e2e run specs/golden/custom-flow-sign-in/complete.e2e.ts +``` + +The test app is built from `integration/templates/expo-native`. `integration/expo-native/bin/control-clerk-expo up --platform ios` builds it as a dev client, leaves it under `.verify/builds/`, and starts the Metro server that the dev client loads its bundle from. The `metro` line that `up` prints has the port. `CLERK_E2E_APP_PATH` is the `.app` or `.apk` in that directory, and `CLERK_E2E_DEV_SERVER` is `http://localhost:`. After `up` ends, no CLI process is running, and Metro stays up until `down`. + +`--video on`, `--retries`, `--grep`, and `--output` are e2e's own flags. e2e writes its report and its screenshots to `.e2e/`, which git ignores. + +| Setting | Needed | What it is | +| ----------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `CLERK_E2E_DEVICE` | yes | The id of a booted simulator, as `xcrun simctl list devices booted` prints it, or the serial of a running emulator, as `adb devices` prints it, such as `emulator-5554`. A name is not accepted. On Android, a `host:port` address and the serial of a phone are refused too. | +| `CLERK_E2E_APP_PATH` | no | A build of the test app for that platform. The first launch of a run installs it. Leave it unset when the app is already on the device. | +| `CLERK_E2E_PLATFORM` | yes | `ios` or `android`. This package drives both, and a run drives one. | +| `CLERK_E2E_DEV_SERVER` | for a dev client | The address of the Metro server that serves the bundle, such as `http://localhost:8081`. Leave it unset for a standalone build, which carries its bundle. | +| `CLERK_PUBLISHABLE_KEY` | yes | The `pk_test_` key of a development instance. | +| `CLERK_SECRET_KEY` | yes | The `sk_test_` key of the same instance. The tests create users and sign-in tickets with it. A `pk_live_` or `sk_live_` key is refused. | + +## The instance + +The tests expect the settings in `src/core/instances/base.json`, which the CLI puts on the application it creates. On an instance with those settings, every test file that has no settings file beside it passes by hand. + +An instance left at Clerk's defaults passes only part of the suite. The tests that sign in with a ticket, or sign up with an email address and a password, pass. The tests that need a phone number, an email code at sign-in, organizations, or a second account on the device fail, because a new instance has none of those turned on. + +Every run creates users with addresses like `verify__+clerk_test@example.com` and never deletes them. A development instance holds 100 users. Use an instance that you can empty. + +## Test files that need the CLI + +This test file has a `.settings.json` file beside it, and passes only on an instance with those settings: + +- `specs/golden/native-modules/biometric-availability.e2e.ts` + +The CLI's `run` puts the instance on the declared settings before the file starts, and e2e alone does not. Run them through the CLI, for example `integration/expo-native/bin/control-clerk-expo run native-modules` from the repo root. + +## What a run by hand does not do + +- It creates no Clerk application, changes no instance settings, and replaces no instance that is filling up. +- It leases and locks no device. Run one test process per device at a time. +- It does not rebuild the app when the sources change. +- It records no video of the whole run. `--video on` records one per attempt. +- It keeps no sealed run directory, scans nothing for secrets, redacts no log, and posts nothing to a pull request. +- It leaves the `agent-device` daemon that e2e started running, and on iOS with it the runner process that the daemon keeps. `npx agent-device daemon stop --clean` stops both. Without `--clean` the iOS runner stays. + +## Layout + +- `e2e.config.ts` and `specs/` are the e2e project. `specs/golden/` has the tests, `specs/fixtures.ts` has the `host` fixture they use, `specs/app.ts` names the test app, and `specs/support/` is the tests' support code. `specs/support/inputs.ts` is the only file that reads the settings above. +- `bin/` and `src/` are the CLI. Under the CLI the test process gets the same settings, with one difference: in place of the secret key it gets the address and token of a stand-in that the CLI runs on this machine. The stand-in forwards the three Backend API calls the tests make, for this run's test users only, so the test process never holds the key. +- `test/` and `testing/` are the unit tests of both. `npm test` runs them, with no device, key, or network. +- `src/core/`, `specs/support/`, `specs/fixtures.ts`, and `e2e.config.ts` are the same files, byte for byte, in clerk-ios, clerk-android, and clerk/javascript. `src/core/MANIFEST` lists them, and the CLI's `doctor` fails when they differ from it. diff --git a/integration/expo-native/bin/control-clerk-expo b/integration/expo-native/bin/control-clerk-expo new file mode 100755 index 00000000000..20743dd3019 --- /dev/null +++ b/integration/expo-native/bin/control-clerk-expo @@ -0,0 +1,7 @@ +#!/usr/bin/env node +import { ensureRuntime } from '../src/core/launch.mjs'; + +await ensureRuntime(); +const { main } = await import('../src/core/cli.ts'); +const { host } = await import('../src/host.ts'); +process.exitCode = await main(process.argv.slice(2), host); diff --git a/integration/expo-native/e2e.config.ts b/integration/expo-native/e2e.config.ts new file mode 100644 index 00000000000..a2cc581db48 --- /dev/null +++ b/integration/expo-native/e2e.config.ts @@ -0,0 +1,5 @@ +import { app } from './specs/app.ts'; +import { composeE2EConfig } from './specs/support/config.ts'; +import { readTarget } from './specs/support/inputs.ts'; + +export default composeE2EConfig(app, readTarget(app, process.env), process.env); diff --git a/integration/expo-native/package-lock.json b/integration/expo-native/package-lock.json new file mode 100644 index 00000000000..b3679103700 --- /dev/null +++ b/integration/expo-native/package-lock.json @@ -0,0 +1,1783 @@ +{ + "name": "verify-clerk-expo", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "verify-clerk-expo", + "devDependencies": { + "@e2e-dev/github": "0.4.0", + "@e2e-dev/mobile": "0.10.0", + "@types/node": "24.19.1", + "ai": "7.0.128", + "e2e": "0.18.0", + "typescript": "7.0.2" + }, + "engines": { + "node": ">=24.8.0 <25" + } + }, + "node_modules/@ai-sdk/gateway": { + "version": "4.0.104", + "resolved": "https://registry.npmjs.org/@ai-sdk/gateway/-/gateway-4.0.104.tgz", + "integrity": "sha512-4lTyt8a4BMfVz4QGo+VO5odakmYMX5kq+xiRgdQGGY7o/jP2WSasp6LvIvYvt/lHzz1MfE+xKTOJhz7mMwjynA==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "4.0.22", + "@ai-sdk/provider-utils": "5.0.54", + "@vercel/oidc": "3.2.0" + }, + "engines": { + "node": ">=22" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/gateway/node_modules/@ai-sdk/provider": { + "version": "4.0.22", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider/-/provider-4.0.22.tgz", + "integrity": "sha512-1Jtmn36VNMOBo9CMUQrPqsR744qtqRloEiXpWCYS8ffPlhS2CrSNmoho5lPjp3R7EnkzBsGI6p51TA7fQJQZdA==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "json-schema": "^0.4.0" + }, + "engines": { + "node": ">=22" + } + }, + "node_modules/@ai-sdk/provider": { + "version": "4.0.21", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider/-/provider-4.0.21.tgz", + "integrity": "sha512-UpbC9C1oht8dhfKPbVXSLRZS3DI8uk8n5v2uMjBPNIYGsC2kL045ywq7D8Hh9KgnH9rP5W/GxQdYtWScRouqlA==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "json-schema": "^0.4.0" + }, + "engines": { + "node": ">=22" + } + }, + "node_modules/@ai-sdk/provider-utils": { + "version": "5.0.54", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider-utils/-/provider-utils-5.0.54.tgz", + "integrity": "sha512-amqDxFnw9+dpVrDACRDGmExUUNOH9C1D2bhg/DqWJnamEOieDTXBBXLHujSby/0MqHdAFksnpE8Pzfqx6ABMBw==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "4.0.22", + "@standard-schema/spec": "^1.1.0", + "@workflow/serde": "4.1.0", + "eventsource-parser": "^3.0.8", + "undici": "^7.29.0" + }, + "engines": { + "node": ">=22" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/@ai-sdk/provider-utils/node_modules/@ai-sdk/provider": { + "version": "4.0.22", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider/-/provider-4.0.22.tgz", + "integrity": "sha512-1Jtmn36VNMOBo9CMUQrPqsR744qtqRloEiXpWCYS8ffPlhS2CrSNmoho5lPjp3R7EnkzBsGI6p51TA7fQJQZdA==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "json-schema": "^0.4.0" + }, + "engines": { + "node": ">=22" + } + }, + "node_modules/@clack/core": { + "version": "1.5.1", + "resolved": "https://registry.npmjs.org/@clack/core/-/core-1.5.1.tgz", + "integrity": "sha512-iHTrHA8MtVuLl2TfZySmcKv1qO2PoyC9Z7pfSDozEuV5vtY3/wcOPKJXlqJ5Oq2Cx5DDGQGAMVx6HZfRRoVEbQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "fast-wrap-ansi": "^0.2.0", + "sisteransi": "^1.0.5" + }, + "engines": { + "node": ">= 20.12.0" + } + }, + "node_modules/@clack/prompts": { + "version": "1.8.1", + "resolved": "https://registry.npmjs.org/@clack/prompts/-/prompts-1.8.1.tgz", + "integrity": "sha512-dlT1m5e/0yUL0kRNcQn7yGLVThkgbB0Ga/1AmfDDC/8ik6AIiSf2QLQO2zPYvefsHP0aFgxO93cVLCCfDp7kzQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@clack/core": "1.5.1", + "fast-string-width": "^3.0.2", + "fast-wrap-ansi": "^0.2.0", + "sisteransi": "^1.0.5" + }, + "engines": { + "node": ">= 20.12.0" + } + }, + "node_modules/@e2e-dev/github": { + "version": "0.4.0", + "resolved": "https://registry.npmjs.org/@e2e-dev/github/-/github-0.4.0.tgz", + "integrity": "sha512-h4F51Ncwk9VWgdPgAJCjXvqQrBd6sQSt3yvpl9XNJkLTOccndr3A6eAfw6iY49KvNOpZicdQ/glDHxs1c4Lm7Q==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": "^22.22.3 || >=24.8.0" + }, + "peerDependencies": { + "e2e": ">=0.15.0 <1" + } + }, + "node_modules/@e2e-dev/mobile": { + "version": "0.10.0", + "resolved": "https://registry.npmjs.org/@e2e-dev/mobile/-/mobile-0.10.0.tgz", + "integrity": "sha512-Oz6ePagl8S/WnCQlfw0WuABpGAgMKK0mTshzuO/p8XEdttwBfO9sJsAWG7PiRF9dwEyU1IkCu8eNdtv/pk3akA==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "agent-device": "0.21.22", + "pngjs": "7.0.0", + "zod": "4.6.1" + }, + "engines": { + "node": "^22.22.3 || >=24.8.0" + }, + "peerDependencies": { + "ai": "^7.0.0", + "e2e": ">=0.15.0 <1" + }, + "peerDependenciesMeta": { + "ai": { + "optional": true + } + } + }, + "node_modules/@jridgewell/gen-mapping": { + "version": "0.3.13", + "resolved": "https://registry.npmjs.org/@jridgewell/gen-mapping/-/gen-mapping-0.3.13.tgz", + "integrity": "sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/sourcemap-codec": "^1.5.0", + "@jridgewell/trace-mapping": "^0.3.24" + } + }, + "node_modules/@jridgewell/remapping": { + "version": "2.3.5", + "resolved": "https://registry.npmjs.org/@jridgewell/remapping/-/remapping-2.3.5.tgz", + "integrity": "sha512-LI9u/+laYG4Ds1TDKSJW2YPrIlcVYOwi2fUC6xB43lueCjgxV4lffOCZCtYFiH6TNOX+tQKXx97T4IKHbhyHEQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/gen-mapping": "^0.3.5", + "@jridgewell/trace-mapping": "^0.3.24" + } + }, + "node_modules/@jridgewell/resolve-uri": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz", + "integrity": "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/@jridgewell/sourcemap-codec": { + "version": "1.6.0", + "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.6.0.tgz", + "integrity": "sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==", + "dev": true, + "license": "MIT" + }, + "node_modules/@jridgewell/trace-mapping": { + "version": "0.3.31", + "resolved": "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.31.tgz", + "integrity": "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/resolve-uri": "^3.1.0", + "@jridgewell/sourcemap-codec": "^1.4.14" + } + }, + "node_modules/@modelcontextprotocol/core": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/@modelcontextprotocol/core/-/core-2.3.0.tgz", + "integrity": "sha512-09BHFaNVBe5pB2AhzCfLj0cqmsKLERvZQ4FblzP90UMGLaXVqdZPByFjURwsUYCmxonUW2yAwgTTvle+8O2/8w==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "zod": "^4.2.0" + }, + "engines": { + "node": ">=20" + } + }, + "node_modules/@modelcontextprotocol/server": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/@modelcontextprotocol/server/-/server-2.3.0.tgz", + "integrity": "sha512-+6b0LdsQLmHsvS7J6sQJ4weTpzUn61fU1JKS+321/+IyDYoHW5CR2Tgqcs6GgAAQvlesCo6yX3xttgmx+XGtJw==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@modelcontextprotocol/core": "2.3.0", + "zod": "^4.2.0" + }, + "engines": { + "node": ">=20" + } + }, + "node_modules/@oxc-parser/binding-android-arm-eabi": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-android-arm-eabi/-/binding-android-arm-eabi-0.152.0.tgz", + "integrity": "sha512-393VtirINkae4XHBTkq/yQxhOwEr4V+r0et37P2fp+sYZSdYgZTRDPo5Um8vBh0QjMZ+RmaMy4PP0qwr4cTMwA==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-android-arm64": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-android-arm64/-/binding-android-arm64-0.152.0.tgz", + "integrity": "sha512-iYFcExATJNZ0rZj0o02THbD1Nc9jAW0nVgIQPee6ejbSASpbZXlIMw8jxtpzaG14JQTWwGSZ6U5dEtRJXE6V3Q==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-darwin-arm64": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-darwin-arm64/-/binding-darwin-arm64-0.152.0.tgz", + "integrity": "sha512-fnU+DfUzi5/rFy0LzmipYRz06PAUUkOS8kCJQ3Eg6oqlunC1XZXoffFqxmdL51RmcbYtUG3P8DVG1U5fckoa2A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-darwin-x64": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-darwin-x64/-/binding-darwin-x64-0.152.0.tgz", + "integrity": "sha512-Y3U2okryWTs7hDld6UhjJwLNf8/do+x4g4C+1U5SiGwrTLwbO83WJYJA8+l9iFSilWBgnfpd/+U76tfeNxUHzA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-freebsd-x64": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-freebsd-x64/-/binding-freebsd-x64-0.152.0.tgz", + "integrity": "sha512-XkhkRrrPffEw+ZLv/d/5pEni0buxinm/E/N6fLbJCunD1DBbeCTtVP23ESV+4CWCWdSz397asks1ViGslpmWiQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-linux-arm-gnueabihf": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-0.152.0.tgz", + "integrity": "sha512-MLzsRRbdfmdzo6JEhbC1yKU9ItfvM8tEf18HM32jCiNYZEz6J3bRBRl3cTkKR0vBdrk9ZVlcARXtwHccs+epDA==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-linux-arm-musleabihf": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm-musleabihf/-/binding-linux-arm-musleabihf-0.152.0.tgz", + "integrity": "sha512-HHerTEJz7Iy10Ji6BGsk4gqG6PU0UzFuBu91jIEOX89FnrI5ECWSEr7BLpkYQhp7U0QFtMnBhNfQKI0hYZK78w==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-linux-arm64-gnu": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-0.152.0.tgz", + "integrity": "sha512-9Bb8GyzZ2uSLzRDZ27usaKokr07mxiBe2dDF5b6bd2janmaJTT1gL51UfP7AU30yFiXrJvcRyFKREAgKwZYs0w==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-linux-arm64-musl": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm64-musl/-/binding-linux-arm64-musl-0.152.0.tgz", + "integrity": "sha512-om7iDMUroI74A/lfRRWY9/Ev3VWrVsrUZ4tyvyh8raBey/sdcfyAD7tYWc+iykAiotnmftyKTpUKFr1QnaBrAQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-linux-ppc64-gnu": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-0.152.0.tgz", + "integrity": "sha512-Lf1OL2lMTqwDMFC6jnRb30B+WYHKV0XT+bPHcFM6xHtNVwxpobiTzUIvKH6a0LWpvfwyru8Yu6zCGHqqFHH+gg==", + "cpu": [ + "ppc64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-linux-riscv64-gnu": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-riscv64-gnu/-/binding-linux-riscv64-gnu-0.152.0.tgz", + "integrity": "sha512-BYWSMQK+YJ9ZILStPcmmiSVq8KEG9Qdq/c4BHPC3rUEq/fEDCVjUIwdj5ZzjgZ9P9is9PKu0tcAXzq1ZXg4urQ==", + "cpu": [ + "riscv64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-linux-riscv64-musl": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-riscv64-musl/-/binding-linux-riscv64-musl-0.152.0.tgz", + "integrity": "sha512-d2LqHNNNqdH185sRK5dPKWZu201JrNu9/CRFuVEuT+ht7n/QaVblEMWT9MALQPrDWYvhG10P+NSKJFquDWZ09w==", + "cpu": [ + "riscv64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-linux-s390x-gnu": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-0.152.0.tgz", + "integrity": "sha512-8gBeHt5rA/8+pKhHl+zLggfa3L/V/L2at96p7ty6pBE7id6lFO19nFUc15u2PcjC4OYeYK465MKy1fhhbpFIGw==", + "cpu": [ + "s390x" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-linux-x64-gnu": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-x64-gnu/-/binding-linux-x64-gnu-0.152.0.tgz", + "integrity": "sha512-ltgrSF6L+yPTVsIdaAyqKq3zTqzj/1VlNbxWZuxAe8flR3sWdyNsKvdXukoO8/ucJbLickhmieRBge+o/e8Ugw==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-linux-x64-musl": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-x64-musl/-/binding-linux-x64-musl-0.152.0.tgz", + "integrity": "sha512-LcMp6BvrBnewSfdkLHoSqidhoAq4syKOTBV5TYHWd+2VnGkfD/nPRRVIzTBt/Lkbt7pfOFDGtkiTntFS0p6sQQ==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-openharmony-arm64": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-openharmony-arm64/-/binding-openharmony-arm64-0.152.0.tgz", + "integrity": "sha512-jbhoMCslVxzRCRNQ8TSHxjYdSKRvk8O6ciLVDkahJvmyHDzBqba/bMQHM6ULKj/rgHNNbSzlIiakdx8SEfkJzA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-win32-arm64-msvc": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-0.152.0.tgz", + "integrity": "sha512-JZYVlhv36mr/nertnRB7vchJECOBb1Lz6lltalILsB+o9M/is+H6BEDRLQKdqA4JKs6NjJszo46+iM5sfwj6jQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-win32-ia32-msvc": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-win32-ia32-msvc/-/binding-win32-ia32-msvc-0.152.0.tgz", + "integrity": "sha512-1X9oganvVsunRH0lH10UU05N+H9atpX6WsOybByUjwnfcd3h/0tYwvuv16XuHglrIUOnxV2TodbLVEwwafd64g==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-parser/binding-win32-x64-msvc": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-parser/binding-win32-x64-msvc/-/binding-win32-x64-msvc-0.152.0.tgz", + "integrity": "sha512-sw0M80/dn9rcpu+Rqqph/C+MjvDDce6MlXmFjsNQtPEymha0eJy3ml6+foSUTqFfpDIDyHZZ4eoJ43IoMe8bJw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-project/runtime": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-project/runtime/-/runtime-0.152.0.tgz", + "integrity": "sha512-fsq/7JVJo8YAmMPnt3Z7akMdMoCokRND5jjawa5DxuKJBEaI0Dl3ZqEXFlLabKoZhlApIexZ7aycXFcjcTcWvw==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-project/types": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.152.0.tgz", + "integrity": "sha512-oM/5rLBm2tPkg0iBgkH/FOeR3PCDpY19GTgAZjMFM8h9WI9VW7cLgzp6nwtarYKmovavIQZ+Fe/RKX/8C8O/Rw==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/oxc-project" + } + }, + "node_modules/@oxc-transform/binding-android-arm-eabi": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-android-arm-eabi/-/binding-android-arm-eabi-0.152.0.tgz", + "integrity": "sha512-GBVYZdUggANq7ndUuE9HBi1jO6d2+oud3cevfQkpqrV7JsDNwQU5UCGa8rZCvzjIvyn6ikQuJu+sj4RvLyN6Uw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-android-arm64": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-android-arm64/-/binding-android-arm64-0.152.0.tgz", + "integrity": "sha512-pqeCLGUnozzyJE9Ed9js2w+V1lHQ43qPImMfgTh7it+a3LJqUjAt+OesJ6pqkPycRFV6b0pUWZm6EkpM8mk7ag==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-darwin-arm64": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-darwin-arm64/-/binding-darwin-arm64-0.152.0.tgz", + "integrity": "sha512-0d68wdepj2bjnDijPdvC/3MnrZcs8hoCp19qnGX2tyxnSIPevgqa9RX5i31ex5JMczht/VGp4aKeV1p12dwzQQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-darwin-x64": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-darwin-x64/-/binding-darwin-x64-0.152.0.tgz", + "integrity": "sha512-jFTRTyZzYhqSrxkaI9hTwClPTXtdqRyPSXUFGLJmStYS+b6qnhbI1ottivWdAg0HJXytyg8YeqgUJWSu2lU/jA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-freebsd-x64": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-freebsd-x64/-/binding-freebsd-x64-0.152.0.tgz", + "integrity": "sha512-L4Y8IfrcaxjK9kK3ddRAyZPa6nzJHn6fgkVIKoBlP53vmCobPL1mtuIdXQThD0VmV6e4aSAJg2An6qkdjeZC8Q==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-linux-arm-gnueabihf": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-0.152.0.tgz", + "integrity": "sha512-GrohqmIKEK/YXCzAGP8XGQrPVBdUHPpvqdhvw+VutpWZofxDOq39Zed+Gqlea2mycXo+/UNeq1194eaAYkqKQQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-linux-arm-musleabihf": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-linux-arm-musleabihf/-/binding-linux-arm-musleabihf-0.152.0.tgz", + "integrity": "sha512-wUI+JoZEs4+GntxjK2YOKtQDG822wv+aGYm1OUZ2+NAgi8ghOTUP3RH/GX8Vq0qJ2Yw3OwKLUIZzufXklQ7pCQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-linux-arm64-gnu": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-0.152.0.tgz", + "integrity": "sha512-EcY52e0BOfynOA/brCInvQO3ZSukbAEc+BsbpRoRdJ4qV10NK0aS/FheBvkj/V3cFidee6QlrUpqgpN4WKV+Sg==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-linux-arm64-musl": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-linux-arm64-musl/-/binding-linux-arm64-musl-0.152.0.tgz", + "integrity": "sha512-L4OGw0UuuyYKAPnUvvW+tvq6nS/tBoolVuPekNJqYnDYfe5Nh1619ImfhP0o0Wz+SG4OQW35ByRWjVf/oK3x0w==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-linux-ppc64-gnu": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-0.152.0.tgz", + "integrity": "sha512-5K9qNjNKlyhBdGkkIhYFonO/H3Yc37miy5cXefTB71aPOv9jv6HQrDuj6AJcBkggbmkeyEJhlFamUbGMsMGBiA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-linux-riscv64-gnu": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-linux-riscv64-gnu/-/binding-linux-riscv64-gnu-0.152.0.tgz", + "integrity": "sha512-WlGvnZhmRItWqjsGd2OBKPan75FbqpF4HPuGL7P6f7Oz1VHm6IatnwCK2MI16QXH7BEs6GRohZHpmgThc0V/uw==", + "cpu": [ + "riscv64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-linux-riscv64-musl": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-linux-riscv64-musl/-/binding-linux-riscv64-musl-0.152.0.tgz", + "integrity": "sha512-TWZX7QGkU4LF9NS+HBUVWsbQXWDuWt58r+L/31gSJ/iHCJw/3jM2yD408IRVjXYX5jFgmsJbxrUlW3xlVeAL3Q==", + "cpu": [ + "riscv64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-linux-s390x-gnu": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-0.152.0.tgz", + "integrity": "sha512-VIP7463d0nIAGFv2VXb4mRf20qrpZg0kNHBw8E3fLuHj0bt4HVHDgsqAgeFWOonWwSwsM415TFjNmHhlaStCLQ==", + "cpu": [ + "s390x" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-linux-x64-gnu": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-linux-x64-gnu/-/binding-linux-x64-gnu-0.152.0.tgz", + "integrity": "sha512-hwnHMVCnGIoS/eMh7T836mJpWim2iCSWGlCS+aZ+SjhEEUvSSUE54ljua7bk25vaj4L84cFnruTCa7HqyaQ2YQ==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-linux-x64-musl": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-linux-x64-musl/-/binding-linux-x64-musl-0.152.0.tgz", + "integrity": "sha512-5NPItMgM9GfIW0SbpOXvEFIijin5VcNEParmBEm9j+RXr8Ko7pZ3a1ydk/Sy+EOvaSTkXMY6TkVyLfczwdudSQ==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-openharmony-arm64": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-openharmony-arm64/-/binding-openharmony-arm64-0.152.0.tgz", + "integrity": "sha512-pNzMq4AhBUV8E/cc41SgYTjzW/AWq1DtEKz1l56Mn1TKfidJwP8sGnIw0EXhD6zKFUlPoqkWfvoe5TiDAYeT+w==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-win32-arm64-msvc": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-0.152.0.tgz", + "integrity": "sha512-PU6cGmVDYo2HHG9ul6vKj189YJPZrcscdWzQakJD6NU4CqvqISbZLUHwz6DVe0v8CbOe2d24zUdipILaRofZ5g==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-win32-ia32-msvc": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-win32-ia32-msvc/-/binding-win32-ia32-msvc-0.152.0.tgz", + "integrity": "sha512-Sdcoky580dPgj9Qp+i4zrCV7NpIYZW6bBrcQvOwyDmNlJQ2iYK/V3Q3FGNHLulrxgXrpzqQwMcLs8KRIg0u0yw==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@oxc-transform/binding-win32-x64-msvc": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/@oxc-transform/binding-win32-x64-msvc/-/binding-win32-x64-msvc-0.152.0.tgz", + "integrity": "sha512-QnN+/Hsig/vI9zoCNVbbqv0GmZWUGqlPg2ZTEyVw/PvgAXd8pRRfN6CxdKEtJTnaVwJFVYaw/5LR6ZWCVBwDeQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@standard-schema/spec": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@standard-schema/spec/-/spec-1.1.0.tgz", + "integrity": "sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/chai": { + "version": "5.2.3", + "resolved": "https://registry.npmjs.org/@types/chai/-/chai-5.2.3.tgz", + "integrity": "sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/deep-eql": "*", + "assertion-error": "^2.0.1" + } + }, + "node_modules/@types/deep-eql": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/@types/deep-eql/-/deep-eql-4.0.2.tgz", + "integrity": "sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/node": { + "version": "24.19.1", + "resolved": "https://registry.npmjs.org/@types/node/-/node-24.19.1.tgz", + "integrity": "sha512-aS3/DG0oM05K0RIXXP+hKjinGG5IgSSVGzswZxW3O0sS3pH4/fycXundUC9XsszgKCk4gHXylTEK6hyFxVxnoQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "undici-types": ">=7.24.0 <7.24.7" + } + }, + "node_modules/@typescript/typescript-aix-ppc64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-aix-ppc64/-/typescript-aix-ppc64-7.0.2.tgz", + "integrity": "sha512-MTKKkWB7p/0E9xi1d1tHtZ5PiLkGEMIq88pK2CubZjOsLtYTLqhgIgi6zepFa+9GHZ6h05NMCkQxGKiPXMxXtQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-darwin-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-darwin-arm64/-/typescript-darwin-arm64-7.0.2.tgz", + "integrity": "sha512-gowzar9MwS/aRWp6f3a4KUqzRjAZjOsmGNCM6LcTgXum+dBfgsBVMN+AgvOCCbguXyick6LJhpBszxMebJ8syA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-darwin-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-darwin-x64/-/typescript-darwin-x64-7.0.2.tgz", + "integrity": "sha512-SZ9xZInqApNlNGc9s0W1VSsktYSOe9cFqNOIqmN1Gs8SmkjKZYFt017G4VwPxASInODuAdbTW7sXiFUf893RgA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-freebsd-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-freebsd-arm64/-/typescript-freebsd-arm64-7.0.2.tgz", + "integrity": "sha512-W5NH4y/J0plIIS5b2xvTEkU7JFxyqdMAOgf+Ilhl0vHQXKO5dZoxd+C/jEtq56c4F3wk71RB4BMRQ2XdI+bwYQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-freebsd-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-freebsd-x64/-/typescript-freebsd-x64-7.0.2.tgz", + "integrity": "sha512-UMGDx5sTpzNw3WiPebH7l90IWfJggEd+egHt/q6p7/Cm3zqoV7VxkGXt+3DxPIw8CcmvAB0j3sVVfbhX+M4Tpw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-arm": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-arm/-/typescript-linux-arm-7.0.2.tgz", + "integrity": "sha512-gffT3xPz9sR7j/YJExkyPntrI0P2EP9XbOyWzth2/Gs0RstK+90RBcO0ncXoXy/beYll1SXw846Nf2zdnEz0QQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-arm64/-/typescript-linux-arm64-7.0.2.tgz", + "integrity": "sha512-Qh4eU4/y3yDjnfjjyPYihMj5/ODIlmt+Bzu17OI+fiSRDW57QmU5SiN63exPRNJPKUzcc1INa1NXdrJ+MqHjUQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-loong64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-loong64/-/typescript-linux-loong64-7.0.2.tgz", + "integrity": "sha512-uEHck9i8hoAzXPiYRib1O7miOnz23SxIeVl6F4LXox+qov1K35jHcEW6VHKvZI+pyvl7fZEP4MCU5LYvIq1GuQ==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-mips64el": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-mips64el/-/typescript-linux-mips64el-7.0.2.tgz", + "integrity": "sha512-R4KvAMnE43W5Qeqb0Ly56O3mWMWIAgsMyz36DCaycd5nbg/9kzm0liw3JocfRqyJY0KPmzFjbswozXyW0DnIYA==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-ppc64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-ppc64/-/typescript-linux-ppc64-7.0.2.tgz", + "integrity": "sha512-DORx5b3sd/4S7eayxm4FQv+A7CrkUIGRaHiwI8oiHTAI1fAPWhF4J0vAlkC8biAlHSVVwxMQ3tjZ2/DVbnQiiA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-riscv64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-riscv64/-/typescript-linux-riscv64-7.0.2.tgz", + "integrity": "sha512-wf0jqEDOjrPRnKwYRyyJDRo11KMbvMFrU+q4zqKyChODBzvlkbhNQfKvLxQCcwTpdDaXSHZTVuh0JoCrKCUMHQ==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-s390x": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-s390x/-/typescript-linux-s390x-7.0.2.tgz", + "integrity": "sha512-IkwJc3L7yhytWd/ewjyxNDfOmswCm9GWMJT/ue/dU4aZNbwZeYAetq42VyLmsmSjvoX7z74X6ZaYCtzAr0EuGw==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-x64/-/typescript-linux-x64-7.0.2.tgz", + "integrity": "sha512-EYdf2cNg7rgCWJnxCdJ+F3V39O8ihb37eHAu1LK8oAFizgTQbPOK7zHHXbPt8rX24COqODXeI3sIf0fCXG7H/A==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-netbsd-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-netbsd-arm64/-/typescript-netbsd-arm64-7.0.2.tgz", + "integrity": "sha512-+polYF4MF04aPpO5FTkHran9yUQDSXqy5GiSDKpsll5jy3l3+g9QLhpf39T+ePtefhXLOGrLl0QIjkQP6VnelA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-netbsd-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-netbsd-x64/-/typescript-netbsd-x64-7.0.2.tgz", + "integrity": "sha512-8YIT0EHM/3dq10ZOVF/A7pc/YSMtbcecct4rWtexrnSCHOPcpC2KTLXfTCR6vDpnSiY12heNb1GiN/wu+T/FyA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-openbsd-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-openbsd-arm64/-/typescript-openbsd-arm64-7.0.2.tgz", + "integrity": "sha512-APT8+ClYnuYm1u9+kgGXoMj2VzWzcymwh2gNSQVySHfkRDGOTVkoWLjCmOQSaO+PoqQ57B0flRp9SA+7GnnkzQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-openbsd-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-openbsd-x64/-/typescript-openbsd-x64-7.0.2.tgz", + "integrity": "sha512-yX7s+Q0Dln0Dt9tEzZsAjXXR/+ytBM7AlglaqyeMPxQszJ1JhlJdZ6jLA+IzldHtflX81em7lDao1xXu+aRRkg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-sunos-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-sunos-x64/-/typescript-sunos-x64-7.0.2.tgz", + "integrity": "sha512-dLJDGaLZ1D4HPQn62u1n8mBDkJREwMsAkCdkwd4Ieqw+x3TUyTsqY0YiBCtE6H6OzzgGk3iuZ3vFWRS+E8/d1g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-win32-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-win32-arm64/-/typescript-win32-arm64-7.0.2.tgz", + "integrity": "sha512-Gyl1Vy6OsWesLzmq+EP0Fb7b4Nid5232AvcA2SFcdYreldpNtYFFofPjnt62y9hQy7VTaZp65ICJjuAQRaVcIQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-win32-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-win32-x64/-/typescript-win32-x64-7.0.2.tgz", + "integrity": "sha512-0BQ3HkAHHlKLSp1qRvf3SUhGpGsDuhB/jgFw75guyqbxJqEaS0Cw/VFO8i2nHglJUzQCRtMMR/IBAKE3ETMC4g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@vercel/oidc": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/@vercel/oidc/-/oidc-3.2.0.tgz", + "integrity": "sha512-UycprH3T6n3jH0k44NHMa7pnFHGu/N05MjojYr+Mc6I7obkoLIJujSWwin1pCvdy/eOxrI/l3uDLQsmcrOb4ug==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">= 20" + } + }, + "node_modules/@vitest/expect": { + "version": "5.0.3", + "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-5.0.3.tgz", + "integrity": "sha512-N/VGY122AqBx25OB4EatqlVu8PBayvm9dQzpJ8kVq/pOvqAhXTE5MAmoSMzdB8klDU9wcgSEUliwgMNMtxPa1w==", + "dev": true, + "license": "MIT", + "dependencies": { + "@standard-schema/spec": "^1.1.0", + "@types/chai": "^5.2.2", + "@vitest/spy": "5.0.3", + "@vitest/utils": "5.0.3", + "chai": "^6.2.2", + "tinyrainbow": "^3.1.1" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/pretty-format": { + "version": "5.0.3", + "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-5.0.3.tgz", + "integrity": "sha512-3hr5eV5z2ff0l2Q1vaKQbB4j3M3tElw8stDokEOTqXJB3Es5VrT8gBRIMY59gGuFp8Y61sWlWFK1aNUH5dHDYw==", + "dev": true, + "license": "MIT", + "dependencies": { + "tinyrainbow": "^3.1.1" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/spy": { + "version": "5.0.3", + "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-5.0.3.tgz", + "integrity": "sha512-XhFysQTB8AZ+P4gMi+Lpo99vg2AZi0qKpaB9yXQl37+CaMEAPO3iH/wGVnSyL5MPERiLezpqTVtrR6UZH5GCXg==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/utils": { + "version": "5.0.3", + "resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-5.0.3.tgz", + "integrity": "sha512-77fDTZv8eu1CzlIfgPU2yrl5MXSUaUAOvgdZEeVm3X3WIxUubIX2SforqS6176lT32HCF6uBVMYVqfHxw7o6eg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/pretty-format": "5.0.3", + "convert-source-map": "^2.0.0", + "tinyrainbow": "^3.1.1" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@workflow/serde": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/@workflow/serde/-/serde-4.1.0.tgz", + "integrity": "sha512-pav4F2BoirECWR7Nf1TKt+2eETcBj7jj4cBefQ8VXQCA6NPkaKeLfj/zMgi+3zYV5ZIBT4GuUiphsj0/b9hPQQ==", + "dev": true, + "license": "Apache-2.0" + }, + "node_modules/agent-device": { + "version": "0.21.22", + "resolved": "https://registry.npmjs.org/agent-device/-/agent-device-0.21.22.tgz", + "integrity": "sha512-ndUh5b+lZ2fRzzY80TTl/KmxR4bh+3FezSH0UGCjPCpN//1trSqtzzNUWunc4woY/vW3DEiPx5xYIwG+fBOXWA==", + "dev": true, + "license": "MIT", + "bin": { + "agent-device": "bin/agent-device.mjs" + }, + "engines": { + "node": ">=22.12" + }, + "peerDependencies": { + "ai": "^6.0.0 || ^7.0.0" + }, + "peerDependenciesMeta": { + "ai": { + "optional": true + } + } + }, + "node_modules/ai": { + "version": "7.0.128", + "resolved": "https://registry.npmjs.org/ai/-/ai-7.0.128.tgz", + "integrity": "sha512-WMKN83DodYtH52ETPHq0jrSwj3wU0aHkhLHXMrwMRabC/aMpIqeWsXJgjkUwrOqkaKHWONdupk/2ttPRvAyFcg==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/gateway": "4.0.104", + "@ai-sdk/provider": "4.0.22", + "@ai-sdk/provider-utils": "5.0.54" + }, + "engines": { + "node": ">=22" + }, + "peerDependencies": { + "zod": "^3.25.76 || ^4.1.8" + } + }, + "node_modules/ai/node_modules/@ai-sdk/provider": { + "version": "4.0.22", + "resolved": "https://registry.npmjs.org/@ai-sdk/provider/-/provider-4.0.22.tgz", + "integrity": "sha512-1Jtmn36VNMOBo9CMUQrPqsR744qtqRloEiXpWCYS8ffPlhS2CrSNmoho5lPjp3R7EnkzBsGI6p51TA7fQJQZdA==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "json-schema": "^0.4.0" + }, + "engines": { + "node": ">=22" + } + }, + "node_modules/assertion-error": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz", + "integrity": "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + } + }, + "node_modules/chai": { + "version": "6.3.0", + "resolved": "https://registry.npmjs.org/chai/-/chai-6.3.0.tgz", + "integrity": "sha512-XWAtwJ6OHO+tj0EKCs0Y2UamnyOxseZWltU4x2U2wh8g4AigdjwvtUjvLP2tqkA/avxHEtzxNaqGq/YGNwckKg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + } + }, + "node_modules/commander": { + "version": "15.0.0", + "resolved": "https://registry.npmjs.org/commander/-/commander-15.0.0.tgz", + "integrity": "sha512-z67u4ZhzCL/Tydu1lJARtEZYWbWaN7oYLHbsuzocr6y4N6WZAagG3RQ4FW61V1/0+jImpj293XfrcYnd1qxtPg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=22.12.0" + } + }, + "node_modules/convert-source-map": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/convert-source-map/-/convert-source-map-2.0.0.tgz", + "integrity": "sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==", + "dev": true, + "license": "MIT" + }, + "node_modules/e2e": { + "version": "0.18.0", + "resolved": "https://registry.npmjs.org/e2e/-/e2e-0.18.0.tgz", + "integrity": "sha512-mVIJtGXYBMLRK12CESeOowuh86nDDBhjfHIuyELuSFbaEmzl4EC4tVhm3T4Y6jidEGnGexZOATzlQ7tM/xw8JA==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@ai-sdk/provider": "4.0.21", + "@clack/prompts": "1.8.1", + "@jridgewell/remapping": "2.3.5", + "@modelcontextprotocol/server": "2.3.0", + "@oxc-project/runtime": "0.152.0", + "@vitest/expect": "5.0.3", + "commander": "15.0.0", + "get-tsconfig": "4.14.3", + "oxc-parser": "0.152.0", + "oxc-transform": "0.152.0", + "picocolors": "1.1.1", + "pngjs": "7.0.0", + "zod": "4.6.1" + }, + "bin": { + "e2e": "dist/cli/bin.js" + }, + "engines": { + "node": "^22.22.3 || >=24.8.0" + }, + "peerDependencies": { + "@ai-sdk/anthropic": "^4.0.0", + "@ai-sdk/google": "^4.0.0", + "@ai-sdk/openai": "^4.0.0", + "@ai-sdk/openai-compatible": "^3.0.0", + "@ai-sdk/xai": "^5.0.0", + "ai": "^7.0.0" + }, + "peerDependenciesMeta": { + "@ai-sdk/anthropic": { + "optional": true + }, + "@ai-sdk/google": { + "optional": true + }, + "@ai-sdk/openai": { + "optional": true + }, + "@ai-sdk/openai-compatible": { + "optional": true + }, + "@ai-sdk/xai": { + "optional": true + }, + "ai": { + "optional": true + } + } + }, + "node_modules/eventsource-parser": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/eventsource-parser/-/eventsource-parser-3.1.1.tgz", + "integrity": "sha512-EKN1vKAMcZ8MlYMpaNuxN6R9yakzH6uajHcHVTqWJzvu5pWw9DyhbP35HH8MVBQ+dZjAfDxk+A8NiR9KWaXiyQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/fast-string-truncated-width": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/fast-string-truncated-width/-/fast-string-truncated-width-3.0.3.tgz", + "integrity": "sha512-0jjjIEL6+0jag3l2XWWizO64/aZVtpiGE3t0Zgqxv0DPuxiMjvB3M24fCyhZUO4KomJQPj3LTSUnDP3GpdwC0g==", + "dev": true, + "license": "MIT" + }, + "node_modules/fast-string-width": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/fast-string-width/-/fast-string-width-3.0.2.tgz", + "integrity": "sha512-gX8LrtNEI5hq8DVUfRQMbr5lpaS4nMIWV+7XEbXk2b8kiQIizgnlr12B4dA3ZEx3308ze0O4Q1R+cHts8kyUJg==", + "dev": true, + "license": "MIT", + "dependencies": { + "fast-string-truncated-width": "^3.0.2" + } + }, + "node_modules/fast-wrap-ansi": { + "version": "0.2.2", + "resolved": "https://registry.npmjs.org/fast-wrap-ansi/-/fast-wrap-ansi-0.2.2.tgz", + "integrity": "sha512-7F2Fl+TjRSenLqlU3UjSH0iyqopqoZIu7eZVpEirP2g1GtWa2G/ecEmBdgz31+Mxr+ELclgg6sokpSFIQiZ02Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "fast-string-width": "^3.0.2" + } + }, + "node_modules/get-tsconfig": { + "version": "4.14.3", + "resolved": "https://registry.npmjs.org/get-tsconfig/-/get-tsconfig-4.14.3.tgz", + "integrity": "sha512-++QEw4DIY7WGoukz+/+A/8dGYPT9l9yIadnmSgZ8Rjr3YVSVDipQSO9CdnJo9ePqFqUUqh+wk9uIaoiAwsiPkA==", + "dev": true, + "license": "MIT", + "dependencies": { + "resolve-pkg-maps": "^1.0.0" + }, + "funding": { + "url": "https://github.com/privatenumber/get-tsconfig?sponsor=1" + } + }, + "node_modules/json-schema": { + "version": "0.4.0", + "resolved": "https://registry.npmjs.org/json-schema/-/json-schema-0.4.0.tgz", + "integrity": "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA==", + "dev": true, + "license": "(AFL-2.1 OR BSD-3-Clause)" + }, + "node_modules/oxc-parser": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/oxc-parser/-/oxc-parser-0.152.0.tgz", + "integrity": "sha512-ZztsZgaCWmPeKYMdoTT5e0oV/oxD1NeLWNR3kAtr/G707cUPYaaozRlVDHLCGnszHJRszN9mJwmPNpqgwloC0A==", + "dev": true, + "license": "MIT", + "dependencies": { + "@oxc-project/types": "^0.152.0" + }, + "engines": { + "node": "^20.19.0 || >=22.12.0" + }, + "funding": { + "url": "https://github.com/sponsors/oxc-project" + }, + "optionalDependencies": { + "@oxc-parser/binding-android-arm-eabi": "0.152.0", + "@oxc-parser/binding-android-arm64": "0.152.0", + "@oxc-parser/binding-darwin-arm64": "0.152.0", + "@oxc-parser/binding-darwin-x64": "0.152.0", + "@oxc-parser/binding-freebsd-x64": "0.152.0", + "@oxc-parser/binding-linux-arm-gnueabihf": "0.152.0", + "@oxc-parser/binding-linux-arm-musleabihf": "0.152.0", + "@oxc-parser/binding-linux-arm64-gnu": "0.152.0", + "@oxc-parser/binding-linux-arm64-musl": "0.152.0", + "@oxc-parser/binding-linux-ppc64-gnu": "0.152.0", + "@oxc-parser/binding-linux-riscv64-gnu": "0.152.0", + "@oxc-parser/binding-linux-riscv64-musl": "0.152.0", + "@oxc-parser/binding-linux-s390x-gnu": "0.152.0", + "@oxc-parser/binding-linux-x64-gnu": "0.152.0", + "@oxc-parser/binding-linux-x64-musl": "0.152.0", + "@oxc-parser/binding-openharmony-arm64": "0.152.0", + "@oxc-parser/binding-win32-arm64-msvc": "0.152.0", + "@oxc-parser/binding-win32-ia32-msvc": "0.152.0", + "@oxc-parser/binding-win32-x64-msvc": "0.152.0" + } + }, + "node_modules/oxc-transform": { + "version": "0.152.0", + "resolved": "https://registry.npmjs.org/oxc-transform/-/oxc-transform-0.152.0.tgz", + "integrity": "sha512-9rs8qImolzWswXGlhmye9Yz9Mce9hOe4Pivb7rSEhF/6VKn1Q122BUe9Mr1O9xQAoBX+SrF6xXfdgxKHeQf+uQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^20.19.0 || >=22.12.0" + }, + "funding": { + "url": "https://github.com/sponsors/oxc-project" + }, + "optionalDependencies": { + "@oxc-transform/binding-android-arm-eabi": "0.152.0", + "@oxc-transform/binding-android-arm64": "0.152.0", + "@oxc-transform/binding-darwin-arm64": "0.152.0", + "@oxc-transform/binding-darwin-x64": "0.152.0", + "@oxc-transform/binding-freebsd-x64": "0.152.0", + "@oxc-transform/binding-linux-arm-gnueabihf": "0.152.0", + "@oxc-transform/binding-linux-arm-musleabihf": "0.152.0", + "@oxc-transform/binding-linux-arm64-gnu": "0.152.0", + "@oxc-transform/binding-linux-arm64-musl": "0.152.0", + "@oxc-transform/binding-linux-ppc64-gnu": "0.152.0", + "@oxc-transform/binding-linux-riscv64-gnu": "0.152.0", + "@oxc-transform/binding-linux-riscv64-musl": "0.152.0", + "@oxc-transform/binding-linux-s390x-gnu": "0.152.0", + "@oxc-transform/binding-linux-x64-gnu": "0.152.0", + "@oxc-transform/binding-linux-x64-musl": "0.152.0", + "@oxc-transform/binding-openharmony-arm64": "0.152.0", + "@oxc-transform/binding-win32-arm64-msvc": "0.152.0", + "@oxc-transform/binding-win32-ia32-msvc": "0.152.0", + "@oxc-transform/binding-win32-x64-msvc": "0.152.0" + } + }, + "node_modules/picocolors": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", + "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true, + "license": "ISC" + }, + "node_modules/pngjs": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/pngjs/-/pngjs-7.0.0.tgz", + "integrity": "sha512-LKWqWJRhstyYo9pGvgor/ivk2w94eSjE3RGVuzLGlr3NmD8bf7RcYGze1mNdEHRP6TRP6rMuDHk5t44hnTRyow==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.19.0" + } + }, + "node_modules/resolve-pkg-maps": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/resolve-pkg-maps/-/resolve-pkg-maps-1.0.0.tgz", + "integrity": "sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/privatenumber/resolve-pkg-maps?sponsor=1" + } + }, + "node_modules/sisteransi": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/sisteransi/-/sisteransi-1.0.5.tgz", + "integrity": "sha512-bLGGlR1QxBcynn2d5YmDX4MGjlZvy2MRBDRNHLJ8VI6l6+9FUiyTFNJ0IveOSP0bcXgVDPRcfGqA0pjaqUpfVg==", + "dev": true, + "license": "MIT" + }, + "node_modules/tinyrainbow": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-3.2.0.tgz", + "integrity": "sha512-LgO3D9yZJjApUiuUfl9iFAwrtaX4+lok3wJIqttGoKCHlWUqHqbQpnxCf82L8FjgKsh4iGo78hqJwgL8F6To2A==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/typescript": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-7.0.2.tgz", + "integrity": "sha512-8FYau96o3NKOhbjKi/qNvG/W5jhzxkbdm5sj9AbZ/5T5sWqn3hJgLfGx27sRKZWTvyzCP8dLRBTf5tBTSRVUNA==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc" + }, + "engines": { + "node": ">=16.20.0" + }, + "optionalDependencies": { + "@typescript/typescript-aix-ppc64": "7.0.2", + "@typescript/typescript-darwin-arm64": "7.0.2", + "@typescript/typescript-darwin-x64": "7.0.2", + "@typescript/typescript-freebsd-arm64": "7.0.2", + "@typescript/typescript-freebsd-x64": "7.0.2", + "@typescript/typescript-linux-arm": "7.0.2", + "@typescript/typescript-linux-arm64": "7.0.2", + "@typescript/typescript-linux-loong64": "7.0.2", + "@typescript/typescript-linux-mips64el": "7.0.2", + "@typescript/typescript-linux-ppc64": "7.0.2", + "@typescript/typescript-linux-riscv64": "7.0.2", + "@typescript/typescript-linux-s390x": "7.0.2", + "@typescript/typescript-linux-x64": "7.0.2", + "@typescript/typescript-netbsd-arm64": "7.0.2", + "@typescript/typescript-netbsd-x64": "7.0.2", + "@typescript/typescript-openbsd-arm64": "7.0.2", + "@typescript/typescript-openbsd-x64": "7.0.2", + "@typescript/typescript-sunos-x64": "7.0.2", + "@typescript/typescript-win32-arm64": "7.0.2", + "@typescript/typescript-win32-x64": "7.0.2" + } + }, + "node_modules/undici": { + "version": "7.30.0", + "resolved": "https://registry.npmjs.org/undici/-/undici-7.30.0.tgz", + "integrity": "sha512-dkrQXeHSaoamnItlYbmzG0wFYrM0ZwDxCIg0A7aKjTyyhh9svRzCNFEzV+Vm05/yehjCzjDZ31KXfGEjYSztDQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=20.18.1" + } + }, + "node_modules/undici-types": { + "version": "7.24.6", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.24.6.tgz", + "integrity": "sha512-WRNW+sJgj5OBN4/0JpHFqtqzhpbnV0GuB+OozA9gCL7a993SmU+1JBZCzLNxYsbMfIeDL+lTsphD5jN5N+n0zg==", + "dev": true, + "license": "MIT" + }, + "node_modules/zod": { + "version": "4.6.1", + "resolved": "https://registry.npmjs.org/zod/-/zod-4.6.1.tgz", + "integrity": "sha512-341aRWQsve0rvronKNTqZpjmzdbUDlFuzHaI/XLg/Ej82qffDJRRfBTCuv7+9q/rMjB6LSLyEBnW4InJeMtt/Q==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/colinhacks" + } + } + } +} diff --git a/integration/expo-native/package.json b/integration/expo-native/package.json new file mode 100644 index 00000000000..72189b7f02e --- /dev/null +++ b/integration/expo-native/package.json @@ -0,0 +1,20 @@ +{ + "name": "verify-clerk-expo", + "private": true, + "type": "module", + "scripts": { + "test": "node --test test/*.test.ts", + "typecheck": "tsc -p ." + }, + "devDependencies": { + "@e2e-dev/github": "0.4.0", + "@e2e-dev/mobile": "0.10.0", + "@types/node": "24.19.1", + "ai": "7.0.128", + "e2e": "0.18.0", + "typescript": "7.0.2" + }, + "engines": { + "node": ">=24.8.0 <25" + } +} diff --git a/integration/expo-native/tsconfig.json b/integration/expo-native/tsconfig.json new file mode 100644 index 00000000000..a4cd3716872 --- /dev/null +++ b/integration/expo-native/tsconfig.json @@ -0,0 +1,16 @@ +{ + "compilerOptions": { + "target": "es2024", + "module": "nodenext", + "moduleResolution": "nodenext", + "strict": true, + "noUncheckedIndexedAccess": true, + "noEmit": true, + "allowImportingTsExtensions": true, + "erasableSyntaxOnly": true, + "verbatimModuleSyntax": true, + "skipLibCheck": true, + "types": ["node"] + }, + "include": ["src/**/*.ts", "specs/**/*.ts", "test/**/*.ts", "testing/**/*.ts", "e2e.config.ts"] +} From c3c7957bf759399c5262cb92d318734d4b49bda4 Mon Sep 17 00:00:00 2001 From: Mike Pitre <12040919+mikepitre@users.noreply.github.com> Date: Wed, 7 Oct 2026 20:30:58 -0400 Subject: [PATCH 2/7] test(expo): add the shared core: the CLI, device leases, the Backend API stand-in, secrets, and evidence Co-Authored-By: Claude Opus 5.5 --- .../expo-native/specs/support/agent.ts | 63 ++ .../expo-native/specs/support/busy-runner.ts | 34 + .../expo-native/specs/support/clerk.ts | 160 +++++ .../expo-native/specs/support/config.ts | 42 ++ .../expo-native/specs/support/device.ts | 57 ++ integration/expo-native/specs/support/fill.ts | 85 +++ .../expo-native/specs/support/inputs.ts | 153 ++++ .../expo-native/specs/support/launch.ts | 134 ++++ .../expo-native/specs/support/secret.ts | 99 +++ .../expo-native/specs/support/tapping.ts | 21 + .../expo-native/specs/support/types.ts | 94 +++ .../expo-native/specs/support/typing.ts | 121 ++++ .../expo-native/specs/support/waiting.ts | 158 ++++ integration/expo-native/src/core/MANIFEST | 36 + integration/expo-native/src/core/agent.ts | 41 ++ integration/expo-native/src/core/broker.ts | 197 +++++ integration/expo-native/src/core/claims.ts | 63 ++ integration/expo-native/src/core/clerk.ts | 82 +++ integration/expo-native/src/core/cli.ts | 352 +++++++++ integration/expo-native/src/core/devices.ts | 210 ++++++ integration/expo-native/src/core/driver.ts | 438 ++++++++++++ integration/expo-native/src/core/e2e.ts | 275 +++++++ integration/expo-native/src/core/evidence.ts | 80 +++ integration/expo-native/src/core/exec.ts | 79 ++ .../expo-native/src/core/github-report.ts | 73 ++ integration/expo-native/src/core/keys.ts | 12 + integration/expo-native/src/core/launch.d.mts | 5 + integration/expo-native/src/core/launch.mjs | 37 + integration/expo-native/src/core/ledgers.ts | 81 +++ integration/expo-native/src/core/manifest.ts | 42 ++ integration/expo-native/src/core/publish.ts | 75 ++ integration/expo-native/src/core/slot.ts | 73 ++ integration/expo-native/src/core/state.ts | 1 + integration/expo-native/src/core/types.ts | 385 ++++++++++ integration/expo-native/src/core/verbs.ts | 675 ++++++++++++++++++ integration/expo-native/src/core/workspace.ts | 232 ++++++ .../expo-native/testing/claim-taker.ts | 5 + integration/expo-native/testing/fake-clerk.ts | 247 +++++++ .../expo-native/testing/fake-instances.ts | 26 + integration/expo-native/testing/git-env.ts | 19 + .../expo-native/testing/lock-holder.ts | 12 + 41 files changed, 5074 insertions(+) create mode 100644 integration/expo-native/specs/support/agent.ts create mode 100644 integration/expo-native/specs/support/busy-runner.ts create mode 100644 integration/expo-native/specs/support/clerk.ts create mode 100644 integration/expo-native/specs/support/config.ts create mode 100644 integration/expo-native/specs/support/device.ts create mode 100644 integration/expo-native/specs/support/fill.ts create mode 100644 integration/expo-native/specs/support/inputs.ts create mode 100644 integration/expo-native/specs/support/launch.ts create mode 100644 integration/expo-native/specs/support/secret.ts create mode 100644 integration/expo-native/specs/support/tapping.ts create mode 100644 integration/expo-native/specs/support/types.ts create mode 100644 integration/expo-native/specs/support/typing.ts create mode 100644 integration/expo-native/specs/support/waiting.ts create mode 100644 integration/expo-native/src/core/MANIFEST create mode 100644 integration/expo-native/src/core/agent.ts create mode 100644 integration/expo-native/src/core/broker.ts create mode 100644 integration/expo-native/src/core/claims.ts create mode 100644 integration/expo-native/src/core/clerk.ts create mode 100644 integration/expo-native/src/core/cli.ts create mode 100644 integration/expo-native/src/core/devices.ts create mode 100644 integration/expo-native/src/core/driver.ts create mode 100644 integration/expo-native/src/core/e2e.ts create mode 100644 integration/expo-native/src/core/evidence.ts create mode 100644 integration/expo-native/src/core/exec.ts create mode 100644 integration/expo-native/src/core/github-report.ts create mode 100644 integration/expo-native/src/core/keys.ts create mode 100644 integration/expo-native/src/core/launch.d.mts create mode 100644 integration/expo-native/src/core/launch.mjs create mode 100644 integration/expo-native/src/core/ledgers.ts create mode 100644 integration/expo-native/src/core/manifest.ts create mode 100644 integration/expo-native/src/core/publish.ts create mode 100644 integration/expo-native/src/core/slot.ts create mode 100644 integration/expo-native/src/core/state.ts create mode 100644 integration/expo-native/src/core/types.ts create mode 100644 integration/expo-native/src/core/verbs.ts create mode 100644 integration/expo-native/src/core/workspace.ts create mode 100644 integration/expo-native/testing/claim-taker.ts create mode 100644 integration/expo-native/testing/fake-clerk.ts create mode 100644 integration/expo-native/testing/fake-instances.ts create mode 100644 integration/expo-native/testing/git-env.ts create mode 100644 integration/expo-native/testing/lock-holder.ts diff --git a/integration/expo-native/specs/support/agent.ts b/integration/expo-native/specs/support/agent.ts new file mode 100644 index 00000000000..9ee11d184ad --- /dev/null +++ b/integration/expo-native/specs/support/agent.ts @@ -0,0 +1,63 @@ +import { readFileSync, statSync } from 'node:fs'; +import { resolve } from 'node:path'; +import { Secret } from './secret.ts'; + +export const DEFAULT_AGENT_MODEL = 'anthropic/claude-haiku-5.5'; +export const BACKUP_AGENT_MODEL = 'openai/gpt-6-luna-fast'; + +type Env = Readonly>; +type GatewayKey = Secret<'ai-gateway-key'>; + +export type AgentCredential = + | { readonly variable: 'AI_GATEWAY_API_KEY'; readonly key: GatewayKey } + | { readonly variable: 'AI_GATEWAY_API_KEY_FILE'; readonly file: string; readonly key: GatewayKey }; + +export interface Agent { + readonly model: string; + readonly backup: string; + readonly credential: AgentCredential; +} + +export class UnusableAgentKey extends Error { + readonly what: string; + readonly fix: string; + constructor(what: string, fix: string) { + super(`${what}; ${fix}`); + this.what = what; + this.fix = fix; + } +} + +export const isSet = (value: string | undefined): value is string => value !== undefined && value.trim() !== ''; + +function keyFromFile(file: string): GatewayKey { + const unusable = (why: string) => new UnusableAgentKey(`AI_GATEWAY_API_KEY_FILE names ${file}, which ${why}`, 'unset AI_GATEWAY_API_KEY_FILE, or point it at a file that holds the Vercel AI Gateway key'); + let mode: number; + try { + mode = statSync(file).mode; + } catch { + throw unusable('does not exist'); + } + if ((mode & 0o077) !== 0) throw new UnusableAgentKey(`${file} can be read by other users of this machine`, `chmod 600 ${file}`); + let value: string; + try { + value = readFileSync(file, 'utf8').trim(); + } catch (error) { + throw unusable(`cannot be read (${(error as NodeJS.ErrnoException).code ?? (error as Error).message})`); + } + if (value === '') throw new UnusableAgentKey(`${file} holds no key`, `put the Vercel AI Gateway key in ${file}, or unset AI_GATEWAY_API_KEY_FILE`); + return new Secret('ai-gateway-key', value); +} + +function credential(env: Env): AgentCredential | null { + if (isSet(env.AI_GATEWAY_API_KEY)) return { variable: 'AI_GATEWAY_API_KEY', key: new Secret('ai-gateway-key', env.AI_GATEWAY_API_KEY.trim()) }; + if (!isSet(env.AI_GATEWAY_API_KEY_FILE)) return null; + const file = resolve(env.AI_GATEWAY_API_KEY_FILE); + return { variable: 'AI_GATEWAY_API_KEY_FILE', file, key: keyFromFile(file) }; +} + +export function readAgent(env: Env): Agent | null { + const found = credential(env); + if (found === null) return null; + return { model: DEFAULT_AGENT_MODEL, backup: BACKUP_AGENT_MODEL, credential: found }; +} diff --git a/integration/expo-native/specs/support/busy-runner.ts b/integration/expo-native/specs/support/busy-runner.ts new file mode 100644 index 00000000000..cd93d1a4a84 --- /dev/null +++ b/integration/expo-native/specs/support/busy-runner.ts @@ -0,0 +1,34 @@ +const STILL_FINISHING = /iOS (?:automation )?runner is still finishing a (?:previous )?command/; + +export const isRunnerBusy = (error: unknown): boolean => error instanceof Error && STILL_FINISHING.test(error.message); + +export interface BusyWait { + waited(): Promise; +} + +export function busyWaits(most: number, wait: () => Promise): BusyWait { + let left = most; + return { + async waited() { + if (left <= 0) return false; + left -= 1; + await wait(); + return true; + }, + }; +} + +export const NO_BUSY_WAIT: BusyWait = busyWaits(0, async () => undefined); + +export async function onceTheRunnerIsFree(step: () => Promise, busy: BusyWait): Promise { + let refusal: unknown; + for (;;) { + try { + return await step(); + } catch (error) { + if (!isRunnerBusy(error)) throw error; + refusal ??= error; + if (!(await busy.waited())) throw refusal; + } + } +} diff --git a/integration/expo-native/specs/support/clerk.ts b/integration/expo-native/specs/support/clerk.ts new file mode 100644 index 00000000000..9627162e96d --- /dev/null +++ b/integration/expo-native/specs/support/clerk.ts @@ -0,0 +1,160 @@ +import { randomBytes } from 'node:crypto'; +import { readFileSync } from 'node:fs'; +import net from 'node:net'; +import type { ClerkAccess, ClerkInstance } from './inputs.ts'; +import { SHORTEST_SECRET, Secret, protect } from './secret.ts'; +import { commandsFor } from './typing.ts'; +import type { RunId, SecretLike, SeedOptions, SeededUser, TestEmail, TestPhone } from './types.ts'; + +export const BACKEND_API_URL = 'https://api.clerk.com/v1'; + +export const BACKEND_CALLS = { + createUser: { method: 'POST', path: '/users' }, + signInToken: { method: 'POST', path: '/sign_in_tokens' }, + usersByPhone: { method: 'GET', path: '/users' }, +} as const; + +export type BackendCall = keyof typeof BACKEND_CALLS; + +export const TICKET_SECONDS = 120; + +// Node gives each address of a host 250 ms to connect before it tries the next, and fails the request when +// all of them miss it. On a network with no IPv6 route that fails about one request in five to api.clerk.com. +export const CONNECT_ATTEMPT_MS = 2_500; +net.setDefaultAutoSelectFamilyAttemptTimeout(CONNECT_ATTEMPT_MS); + +const notTestIdentity = (value: string, kind: string): Error => new Error(`${value} is not a ${kind}; use an address with +clerk_test or a 555-0100..0199 phone that this run created`); + +export const runEmailPrefix = (run: RunId): string => `verify_${run.toLowerCase().replace(/[^a-z0-9]/g, '_')}_`; + +export function newTestEmail(run: RunId): TestEmail { + return parseTestEmail(`${runEmailPrefix(run)}${randomBytes(4).toString('hex')}+clerk_test@example.com`); +} + +const TEST_EMAIL = /^[a-z0-9._-]+\+clerk_test@[a-z0-9.-]+\.[a-z]+$/; +export function parseTestEmail(value: string): TestEmail { + if (!TEST_EMAIL.test(value)) throw notTestIdentity(value, '+clerk_test email'); + return value as TestEmail; +} + +export function parseTestPhone(value: string): TestPhone { + const digits = value.replace(/[^0-9]/g, ''); + const national = digits.length === 11 && digits.startsWith('1') ? digits.slice(1) : digits; + if (!/^[2-9]\d{2}55501\d{2}$/.test(national)) throw notTestIdentity(value, '555-0100..0199 test phone'); + return `+1${national}` as TestPhone; +} + +export const TEST_PHONES: readonly TestPhone[] = Array.from({ length: 100 }, (_, n) => parseTestPhone(`+1201555${String(100 + n).padStart(4, '0')}`)); + +export function typedPassword(value: string): Secret<'password'> { + const password = new Secret('password', value); + for (const typedAtOnce of commandsFor(value)) if (typedAtOnce.length >= SHORTEST_SECRET) protect(typedAtOnce); + return password; +} + +export function newTestPassword(): Secret<'password'> { + return typedPassword(`${randomBytes(12).toString('hex')}Aa1!`); +} + +export const runPassword = (run: RunId): Secret<'password'> => typedPassword(`Verify-${run}-Pw1!`); + +export interface TestUsers { + newEmail(): TestEmail; + newPhone(): Promise; + seed(options?: SeedOptions): Promise; + signInTicket(user: SeededUser): Promise>; +} + +interface Reply { + readonly status: number; + readonly json: unknown; +} + +const ok = (reply: Reply): boolean => reply.status >= 200 && reply.status < 300; + +const errorsOf = (reply: Reply): readonly { readonly code?: string; readonly message?: string; readonly long_message?: string }[] => (reply.json as { errors?: { code?: string; message?: string; long_message?: string }[] } | null)?.errors ?? []; + +const phoneIsTaken = (reply: Reply): boolean => reply.status === 422 && errorsOf(reply).some((error) => error.code === 'form_identifier_exists'); + +function refusal(call: BackendCall, reply: Reply): Error { + const said = errorsOf(reply).map((error) => error.long_message ?? error.message ?? error.code ?? 'no reason given').join('; '); + const { method, path } = BACKEND_CALLS[call]; + return new Error(`Clerk ${method} ${path} answered ${reply.status}${said === '' ? '' : `: ${said}`}`); +} + +function endpoint(access: ClerkAccess): { readonly baseUrl: string; readonly bearer: SecretLike } { + if (access.kind === 'secret-key') return { baseUrl: BACKEND_API_URL, bearer: access.key }; + return { baseUrl: access.url, bearer: new Secret('stand-in-token', readFileSync(access.tokenFile, 'utf8')) }; +} + +export function testUsers(clerk: ClerkInstance, run: RunId, fetchImpl: typeof fetch = fetch): TestUsers { + runPassword(run); + const handedOut = new Set(); + + async function ask(call: BackendCall, sent: { readonly query?: string; readonly body?: unknown }): Promise { + const { method, path } = BACKEND_CALLS[call]; + const { baseUrl, bearer } = endpoint(clerk.access); + const response = await bearer.use('bapi-authorization', (plain) => + fetchImpl(`${baseUrl}${path}${sent.query === undefined ? '' : `?${sent.query}`}`, { + method, + headers: { Authorization: `Bearer ${plain}`, 'Content-Type': 'application/json' }, + ...(sent.body === undefined ? {} : { body: JSON.stringify(sent.body) }), + signal: AbortSignal.timeout(30_000), + }), + ); + const text = await response.text(); + try { + return { status: response.status, json: text.length > 0 ? JSON.parse(text) : null }; + } catch { + return { status: response.status, json: null }; + } + } + + function phonesNotHandedOut(): readonly TestPhone[] { + const first = Math.floor(Math.random() * TEST_PHONES.length); + return TEST_PHONES.map((_, i) => TEST_PHONES[(first + i) % TEST_PHONES.length]!).filter((phone) => !handedOut.has(phone)); + } + + const everyPhoneTaken = (otherwise: string): Error => new Error(`every 555-0100..0199 test phone is taken on this instance; delete its test users${otherwise}`); + + return { + newEmail: () => newTestEmail(run), + async newPhone() { + for (const phone of phonesNotHandedOut()) { + const holders = await ask('usersByPhone', { query: `phone_number=${encodeURIComponent(phone)}` }); + if (!ok(holders)) throw refusal('usersByPhone', holders); + if (Array.isArray(holders.json) && holders.json.length > 0) continue; + handedOut.add(phone); + return phone; + } + throw everyPhoneTaken(''); + }, + async seed(options = {}) { + const email = newTestEmail(run); + const password = options.password === true ? newTestPassword() : null; + for (const phone of options.phone === true ? phonesNotHandedOut() : [null]) { + const created = await ask('createUser', { + body: { + email_address: [email], + ...(phone === null ? {} : { phone_number: [phone] }), + ...(password === null ? { skip_password_requirement: true } : { password: password.use('bapi-user-password', (plain) => plain), bypass_client_trust: true }), + }, + }); + if (phone !== null && phoneIsTaken(created)) continue; + if (!ok(created)) throw refusal('createUser', created); + const id = (created.json as { id?: unknown } | null)?.id; + if (typeof id !== 'string') throw new Error('Clerk created a user without an id'); + if (phone !== null) handedOut.add(phone); + return { id, email, phone, password }; + } + throw everyPhoneTaken(', or seed without a phone'); + }, + async signInTicket(user) { + const minted = await ask('signInToken', { body: { user_id: user.id, expires_in_seconds: TICKET_SECONDS } }); + if (!ok(minted)) throw refusal('signInToken', minted); + const token = (minted.json as { token?: unknown } | null)?.token; + if (typeof token !== 'string') throw new Error('Clerk returned a sign-in token without a token'); + return new Secret('ticket', token); + }, + }; +} diff --git a/integration/expo-native/specs/support/config.ts b/integration/expo-native/specs/support/config.ts new file mode 100644 index 00000000000..7a9f183753a --- /dev/null +++ b/integration/expo-native/specs/support/config.ts @@ -0,0 +1,42 @@ +import { createRequire } from 'node:module'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import type { E2EConfig } from 'e2e'; +import { mobile } from '@e2e-dev/mobile'; +import { readAgent } from './agent.ts'; +import type { Target, TestApp } from './inputs.ts'; + +export const ASSERTION_TIMEOUT_MS = 10_000; +export const TEST_TIMEOUT_MS = 240_000; + +export const PACKAGE_DIR = join(dirname(fileURLToPath(import.meta.url)), '..', '..'); + +type Env = Readonly>; + +const requireOnlyForAnAgent = createRequire(import.meta.url); + +function agentConfig(env: Env): Pick { + const agent = readAgent(env); + if (agent === null) return {}; + const { createGateway } = requireOnlyForAnAgent('ai') as typeof import('ai'); + const gateway = agent.credential.key.use('gateway-provider', (apiKey) => createGateway({ apiKey })); + return { agents: { default: { model: gateway(agent.model), providerOptions: { gateway: { models: [agent.backup] } } } }, cache: 'off' }; +} + +export function composeE2EConfig(app: TestApp, target: Target, env: Env): E2EConfig { + return { + tests: ['specs/**/*.e2e.ts'], + targets: [ + { + name: target.platform, + engine: mobile({ platform: target.platform, device: [target.device.id], session: target.session, videoTouches: false }), + app: { bundleId: app.id(target.platform), appPath: target.build.path ?? undefined }, + }, + ], + workers: 1, + timeout: TEST_TIMEOUT_MS, + assertionTimeout: ASSERTION_TIMEOUT_MS, + trace: 'off', + ...agentConfig(env), + }; +} diff --git a/integration/expo-native/specs/support/device.ts b/integration/expo-native/specs/support/device.ts new file mode 100644 index 00000000000..320a1541048 --- /dev/null +++ b/integration/expo-native/specs/support/device.ts @@ -0,0 +1,57 @@ +import { execFile } from 'node:child_process'; +import { existsSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { join } from 'node:path'; +import type { Device } from './inputs.ts'; +import type { Platform } from './types.ts'; + +export interface CommandLine { + readonly command: string; + readonly args: readonly string[]; +} + +export interface CommandResult { + readonly code: number; + readonly stdout: string; + readonly stderr: string; +} + +export type CommandRunner = (command: string, args: readonly string[]) => Promise; + +export function agentDevice(device: Device, platform: Platform): { readonly selector: readonly string[]; readonly env: Readonly> } { + return { selector: ['--platform', platform, platform === 'ios' ? '--udid' : '--serial', device.id], env: {} }; +} + +const REVERSE_FLAGS: ReadonlySet = new Set(['--list', '--no-rebind', '--remove', '--remove-all']); + +export function deviceToolCommand(platform: Platform, deviceId: string, args: readonly string[], adb = 'adb'): CommandLine | null { + if (platform !== 'android') return null; + const [subcommand, ...rest] = args; + const allowed = subcommand === 'shell' || (subcommand === 'reverse' && rest.every((arg) => REVERSE_FLAGS.has(arg) || /^tcp:\d{1,5}$/.test(arg))); + return allowed ? { command: adb, args: ['-s', deviceId, ...args] } : null; +} + +function localAdb(env: Readonly>, home: string): string { + const byDefault = process.platform === 'darwin' ? join(home, 'Library', 'Android', 'sdk') : join(home, 'Android', 'Sdk'); + const roots = [env.ANDROID_HOME, env.ANDROID_SDK_ROOT, byDefault].filter((root): root is string => root !== undefined && root !== ''); + return roots.map((root) => join(root, 'platform-tools', 'adb')).find((adb) => existsSync(adb)) ?? 'adb'; +} + +const run: CommandRunner = (command, args) => + new Promise((resolve) => { + execFile(command, [...args], (error, stdout, stderr) => { + if (error === null) resolve({ code: 0, stdout, stderr }); + else resolve(typeof error.code === 'number' ? { code: error.code, stdout, stderr } : { code: 127, stdout, stderr: stderr + error.message }); + }); + }); + +export async function deviceCommand( + device: Device, + platform: Platform, + args: readonly string[], + options: { readonly runner?: CommandRunner; readonly env?: Readonly>; readonly home?: string } = {}, +): Promise { + const local = deviceToolCommand(platform, device.id, args, localAdb(options.env ?? process.env, options.home ?? homedir())); + if (local === null) throw new Error(`no ${platform} device command starts with ${args[0] ?? 'nothing'}; on Android use adb shell or adb reverse arguments, and on iOS pass what the app needs as launch arguments`); + return (options.runner ?? run)(local.command, local.args); +} diff --git a/integration/expo-native/specs/support/fill.ts b/integration/expo-native/specs/support/fill.ts new file mode 100644 index 00000000000..024a88699cd --- /dev/null +++ b/integration/expo-native/specs/support/fill.ts @@ -0,0 +1,85 @@ +import { onceTheRunnerIsFree, type BusyWait } from './busy-runner.ts'; +import { tapOnceUncovered } from './tapping.ts'; +import { typeConfirmed } from './typing.ts'; +import type { Platform } from './types.ts'; + +interface Frame { + readonly x: number; + readonly y: number; + readonly width: number; + readonly height: number; +} + +export interface FieldNode { + inputValue(): Promise; + boundingBox(): Promise; +} + +export interface NamedField { + tap(options: { readonly timeout: number }): Promise; + all(): Promise; +} + +export interface FocusedInput extends FieldNode { + count(): Promise; + tap(options: { readonly timeout: number }): Promise; +} + +export interface FillDevice { + readonly platform: Platform; + readonly focused: FocusedInput; + readonly texts: { allTextContents(): Promise }; + type(command: readonly [string, ...string[]]): Promise; +} + +export interface FillPace { + readonly tapTimeoutMs: number; + readonly reads: number; + now(): number; + wait(): Promise; +} + +export async function fillField(field: NamedField, text: string, device: FillDevice, pace: FillPace, busy: BusyWait): Promise { + const { focused } = device; + const tap = (): Promise => tapOnceUncovered(() => field.tap({ timeout: pace.tapTimeoutMs }), { timeoutMs: pace.tapTimeoutMs, now: pace.now, wait: pace.wait }, busy); + await tap(); + const read = (reading: () => Promise, unreadable: T): Promise => onceTheRunnerIsFree(reading, busy).catch(() => unreadable); + const frame = (): Promise => read(async () => ((await focused.count()) === 1 ? await focused.boundingBox() : null), null); + const wordsShown = (): Promise => read(async () => (await device.texts.allTextContents()).map((text) => text.replace(/\P{L}+/gu, '')), null); + const input = await frame(); + const wordsBeforeTyping = await wordsShown(); + let namedInput: FieldNode | null = null; + await typeConfirmed( + { + type: (value) => device.type(['type', value]), + async valueIfReadable() { + const now = await frame(); + if (input === null || now === null || now.x !== input.x || now.y !== input.y || now.width !== input.width || now.height !== input.height) return null; + const value = await read(() => focused.inputValue(), null); + if (value !== '') return value; + const words = await wordsShown(); + return wordsBeforeTyping !== null && words !== null && words.every((shown) => wordsBeforeTyping.includes(shown)) ? '' : null; + }, + valuesOfTheNamedNodes: () => + read(async () => { + const values: string[] = []; + for (const node of await field.all()) { + const value = await node.inputValue(); + if (value !== '') namedInput = node; + values.push(value); + } + return values; + }, []), + refocus: () => onceTheRunnerIsFree(() => focused.tap({ timeout: pace.tapTimeoutMs }), busy), + tapAgain: tap, + async replace(value) { + const box = input ?? (await namedInput?.boundingBox().catch(() => null)) ?? null; + if (box !== null) await device.type(['fill', String(Math.round(box.x + box.width / 2)), String(Math.round(box.y + box.height / 2)), value]); + }, + ...(device.platform === 'android' ? { nothingFocused: async () => (await focused.count()) !== 1 } : {}), + }, + text, + { reads: pace.reads, wait: pace.wait }, + busy, + ); +} diff --git a/integration/expo-native/specs/support/inputs.ts b/integration/expo-native/specs/support/inputs.ts new file mode 100644 index 00000000000..a32decd753b --- /dev/null +++ b/integration/expo-native/specs/support/inputs.ts @@ -0,0 +1,153 @@ +import { randomBytes } from 'node:crypto'; +import { Secret } from './secret.ts'; +import type { AppEntry, Platform, PublishableKey, RunId } from './types.ts'; + +type Env = Readonly>; + +export interface Device { + readonly kind: 'local'; + readonly id: string; +} + +export interface AppBuild { + readonly path: string | null; + readonly devServer: string | null; +} + +export interface Target { + readonly platform: Platform; + readonly device: Device; + readonly session: string; + readonly build: AppBuild; +} + +export type ClerkAccess = + | { readonly kind: 'secret-key'; readonly key: Secret<'clerk-secret-key'> } + | { readonly kind: 'stand-in'; readonly url: string; readonly tokenFile: string }; + +export interface ClerkInstance { + readonly publishableKey: PublishableKey; + readonly access: ClerkAccess; +} + +export interface Inputs { + readonly target: Target; + readonly clerk: ClerkInstance; + readonly run: RunId; +} + +export interface TestApp { + readonly platforms: readonly [Platform, ...Platform[]]; + id(platform: Platform): string; + entry(platform: Platform, devServer: string | null): AppEntry; +} + +const NAMES = { + platform: 'CLERK_E2E_PLATFORM', + device: 'CLERK_E2E_DEVICE', + session: 'CLERK_E2E_DEVICE_SESSION', + appPath: 'CLERK_E2E_APP_PATH', + devServer: 'CLERK_E2E_DEV_SERVER', + publishableKey: 'CLERK_PUBLISHABLE_KEY', + secretKey: 'CLERK_SECRET_KEY', + apiUrl: 'CLERK_E2E_API_URL', + apiTokenFile: 'CLERK_E2E_API_TOKEN_FILE', + run: 'CLERK_E2E_RUN_ID', +} as const; + +export const INPUT_VARIABLES: readonly string[] = Object.values(NAMES); + +const given = (env: Env, name: string): string | null => { + const value = env[name]?.trim(); + return value === undefined || value === '' ? null : value; +}; + +const DEVICES: Readonly> = { + ios: { id: /^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$/i, expects: 'the UDID of a simulator; a name is not accepted', listedBy: 'xcrun simctl list devices booted' }, + android: { id: /^emulator-\d+$/, expects: 'the emulator- serial of an emulator; an AVD name, a host:port address, and the serial of a phone are not accepted', listedBy: 'adb devices' }, +}; + +function readPlatform(app: TestApp, env: Env): Platform { + const named = given(env, NAMES.platform); + const [only, ...others] = app.platforms; + if (named === null) { + if (others.length === 0) return only; + throw new Error(`${NAMES.platform} is not set, and this test app runs on ${app.platforms.join(' and ')}; set it to one of them`); + } + const platform = app.platforms.find((candidate) => candidate === named); + if (platform === undefined) throw new Error(`${NAMES.platform} is ${named}, and this test app runs on ${app.platforms.join(' and ')}`); + return platform; +} + +function readDevice(platform: Platform, env: Env): Device { + const id = given(env, NAMES.device); + const { id: shape, expects, listedBy } = DEVICES[platform]; + if (id === null) throw new Error(`${NAMES.device} is not set; set it to the id of a booted device, which \`${listedBy}\` prints`); + if (!shape.test(id)) throw new Error(`${NAMES.device} is ${id}, which is not ${expects}, so copy the id that \`${listedBy}\` prints`); + return { kind: 'local', id }; +} + +export function readTarget(app: TestApp, env: Env): Target { + const platform = readPlatform(app, env); + return { + platform, + device: readDevice(platform, env), + session: given(env, NAMES.session) ?? `clerk-e2e-${platform}`, + build: { path: given(env, NAMES.appPath), devServer: given(env, NAMES.devServer) }, + }; +} + +const LOOPBACK = /^http:\/\/127\.0\.0\.1:\d{1,5}(\/|$)/; + +export function readClerk(env: Env): ClerkInstance { + const publishableKey = given(env, NAMES.publishableKey); + if (publishableKey === null) throw new Error(`${NAMES.publishableKey} is not set; set it to the publishable key of a development instance`); + if (!publishableKey.startsWith('pk_test_')) throw new Error(`${NAMES.publishableKey} is not a pk_test_ key; these tests create and sign in users, so they run against a development instance only`); + const secretKey = given(env, NAMES.secretKey); + const url = given(env, NAMES.apiUrl); + if (secretKey !== null && url !== null) throw new Error(`${NAMES.secretKey} and ${NAMES.apiUrl} are both set; set one, so there is one way to reach Clerk`); + if (secretKey !== null) { + if (!secretKey.startsWith('sk_test_')) throw new Error(`${NAMES.secretKey} is not an sk_test_ key; these tests create and sign in users, so they run against a development instance only`); + return { publishableKey: publishableKey as PublishableKey, access: { kind: 'secret-key', key: new Secret('clerk-secret-key', secretKey) } }; + } + if (url === null) throw new Error(`${NAMES.secretKey} is not set; set it to the secret key of the same development instance as ${NAMES.publishableKey}`); + if (!LOOPBACK.test(url)) throw new Error(`${NAMES.apiUrl} is ${url}, which is not an http://127.0.0.1: address; only a stand-in on this machine may take the place of Clerk's Backend API`); + const tokenFile = given(env, NAMES.apiTokenFile); + if (tokenFile === null) throw new Error(`${NAMES.apiUrl} is set and ${NAMES.apiTokenFile} is not; the stand-in at that address needs its token file`); + return { publishableKey: publishableKey as PublishableKey, access: { kind: 'stand-in', url, tokenFile } }; +} + +const RUN_ID = /^r\d{8}-\d{6}-[0-9a-f]{4}$/; + +export const isRunId = (value: string): value is RunId => RUN_ID.test(value); + +export function newRunId(): RunId { + const now = new Date(); + const pad = (n: number) => String(n).padStart(2, '0'); + const date = `${now.getFullYear()}${pad(now.getMonth() + 1)}${pad(now.getDate())}`; + const time = `${pad(now.getHours())}${pad(now.getMinutes())}${pad(now.getSeconds())}`; + return `r${date}-${time}-${randomBytes(2).toString('hex')}` as RunId; +} + +export function readRun(env: Env): RunId { + const named = given(env, NAMES.run); + if (named === null) return newRunId(); + if (!isRunId(named)) throw new Error(`${NAMES.run} is ${named}, which is not a run id like r20261002-141210-7c1e; unset it and the tests make one`); + return named; +} + +export function inputsEnv(inputs: Inputs): Readonly> { + const { target, clerk } = inputs; + return { + [NAMES.platform]: target.platform, + [NAMES.device]: target.device.id, + [NAMES.session]: target.session, + ...(target.build.path === null ? {} : { [NAMES.appPath]: target.build.path }), + ...(target.build.devServer === null ? {} : { [NAMES.devServer]: target.build.devServer }), + [NAMES.publishableKey]: clerk.publishableKey, + ...(clerk.access.kind === 'secret-key' + ? { [NAMES.secretKey]: clerk.access.key.use('test-process-environment', (plain) => plain) } + : { [NAMES.apiUrl]: clerk.access.url, [NAMES.apiTokenFile]: clerk.access.tokenFile }), + [NAMES.run]: inputs.run, + }; +} diff --git a/integration/expo-native/specs/support/launch.ts b/integration/expo-native/specs/support/launch.ts new file mode 100644 index 00000000000..f030b32d958 --- /dev/null +++ b/integration/expo-native/specs/support/launch.ts @@ -0,0 +1,134 @@ +import { redact } from './secret.ts'; +import type { AppEntry, HostLaunch, LaunchId, LaunchOptions, Platform, PublishableKey, RunId, SecretLike, SeededUser, StorageScope } from './types.ts'; + +const PLIST_LEADERS = ['(', '{', '<', '"']; + +function checkedValue(key: string, value: string): string { + if (value.length === 0) throw new Error(`launch input ${key} is empty; pass a non-empty value`); + if (PLIST_LEADERS.some((leader) => value.startsWith(leader))) { + throw new Error(`launch input ${key} starts with ${value[0]}, which iOS would parse as a property list; pass a plain string value`); + } + return value; +} + +export function encodeLaunchArguments(platform: Platform, launch: HostLaunch): readonly string[] { + const pairs: [string, string][] = [ + ['verifyPublishableKey', launch.verifyPublishableKey], + ['verifyRunId', launch.verifyRunId], + ['verifyStorageScope', launch.verifyStorageScope], + ['verifyLaunchId', launch.verifyLaunchId], + ]; + if (launch.verifyAuthMode !== undefined) pairs.push(['verifyAuthMode', launch.verifyAuthMode]); + if (launch.verifyInitialIdentifier !== undefined) pairs.push(['verifyInitialIdentifier', launch.verifyInitialIdentifier]); + if (launch.verifyLogLevel !== undefined) pairs.push(['verifyLogLevel', launch.verifyLogLevel]); + const ticket = launch.verifySignInTicket?.use('launch-argument', (plain) => plain); + if (ticket !== undefined) pairs.push(['verifySignInTicket', ticket]); + return pairs.flatMap(([key, value]) => { + const checked = checkedValue(key, value); + switch (platform) { + case 'ios': + return [`-${key}`, checked]; + case 'android': + return ['--es', key, checked]; + default: { + const exhaustive: never = platform; + return exhaustive; + } + } + }); +} + +export type AppStart = + | { readonly kind: 'open-app'; readonly launchArguments: readonly string[] } + | { readonly kind: 'adb'; readonly commands: readonly (readonly string[])[] }; + +function shellQuote(value: string): string { + return `'${value.replaceAll("'", `'\\''`)}'`; +} + +export function appStart(platform: Platform, appId: string, entry: AppEntry, launchArguments: readonly string[]): AppStart { + if (entry.kind === 'binary') return { kind: 'open-app', launchArguments }; + const all = [...entry.launchArguments, ...launchArguments]; + if (platform === 'ios') { + if (entry.openLink !== null) { + throw new Error('a dev-client entry on iOS cannot use openLink yet; pass the URL as a launch argument in entry.launchArguments, such as --initialUrl '); + } + return { kind: 'open-app', launchArguments: all }; + } + if (entry.androidActivity === null) { + throw new Error('a dev-client entry on Android needs androidActivity; set androidActivity in the entry specs/app.ts returns'); + } + const start = ['am', 'start', '-W', '-n', `${appId}/${entry.androidActivity}`, ...(entry.openLink === null ? [] : ['-d', entry.openLink]), ...all]; + return { + kind: 'adb', + commands: [ + ['shell', `am force-stop ${shellQuote(appId)}`], + ['shell', start.map(shellQuote).join(' ')], + ], + }; +} + +export interface AppStartDriver { + openApp(appId: string, options?: { readonly relaunch: true; readonly launchArguments: readonly string[] }): Promise; + adb(args: readonly string[]): Promise; +} + +const messageOf = (error: unknown): string => redact(error instanceof Error ? error.message : String(error)); + +async function openTwiceAtMost(open: () => Promise): Promise { + try { + await open(); + } catch (first) { + await open().catch((second: unknown) => { + throw new Error(`the app did not open in two tries. The first failed with: ${messageOf(first)}. The second failed with: ${messageOf(second)}`, { cause: second }); + }); + } +} + +export async function performAppStart(start: AppStart, appId: string, driver: AppStartDriver): Promise { + if (start.kind === 'open-app') { + await openTwiceAtMost(() => driver.openApp(appId, { relaunch: true, launchArguments: start.launchArguments })); + return; + } + for (const command of start.commands) await driver.adb(command); + await openTwiceAtMost(() => driver.openApp(appId)); +} + +export interface LaunchedApp { + readonly platform: Platform; + readonly id: string; + readonly entry: AppEntry; + readonly buildPath: string | null; + readonly publishableKey: PublishableKey; + readonly run: RunId; +} + +export interface LaunchDriver extends AppStartDriver { + installApp(path: string): Promise; + signInTicket(user: SeededUser): Promise; +} + +export function appLauncher(app: LaunchedApp, driver: LaunchDriver, installedBuilds: Set, newId: () => string): (options: LaunchOptions) => Promise { + let scope: StorageScope | null = null; + return async (options) => { + if (app.buildPath !== null && !installedBuilds.has(app.buildPath)) { + await driver.installApp(app.buildPath); + installedBuilds.add(app.buildPath); + } + const ticket = options.signedInAs === undefined ? undefined : await driver.signInTicket(options.signedInAs); + scope = options.keepStorage === true && scope !== null ? scope : (newId() as StorageScope); + const launchId = newId() as LaunchId; + const launchArguments = encodeLaunchArguments(app.platform, { + verifyPublishableKey: app.publishableKey, + verifyRunId: app.run, + verifyStorageScope: scope, + verifyLaunchId: launchId, + ...(options.authMode === undefined ? {} : { verifyAuthMode: options.authMode }), + ...(options.initialIdentifier === undefined ? {} : { verifyInitialIdentifier: options.initialIdentifier }), + ...(options.debugLogs === true ? { verifyLogLevel: 'debug' as const } : {}), + ...(ticket === undefined ? {} : { verifySignInTicket: ticket }), + }); + await performAppStart(appStart(app.platform, app.id, app.entry, launchArguments), app.id, driver); + return launchId; + }; +} diff --git a/integration/expo-native/specs/support/secret.ts b/integration/expo-native/specs/support/secret.ts new file mode 100644 index 00000000000..f0b48110922 --- /dev/null +++ b/integration/expo-native/specs/support/secret.ts @@ -0,0 +1,99 @@ +import type { SecretSink } from './types.ts'; + +const knownValues = new Set(); + +export function usedSecretValues(): readonly string[] { + return [...knownValues]; +} + +export function protect(value: string): void { + knownValues.add(value); +} + +export const SHORTEST_SECRET = 8; + +const JWT_START = 'eyJ'; +const SHORTEST_JWT_PART = 13; +const SHORTEST_JWT_SIGNATURE = 10; +const SHORTEST_JWT = 2 * SHORTEST_JWT_PART + SHORTEST_JWT_SIGNATURE + 2; + +type Span = readonly [number, number]; + +const inAJwt = (code: number): boolean => (code >= 48 && code <= 57) || (code >= 65 && code <= 90) || (code >= 97 && code <= 122) || code === 45 || code === 46 || code === 95; + +function jwtsInRun(run: string, offset: number): readonly Span[] { + const found: Span[] = []; + const parts = run.split('.'); + let at = offset; + for (let n = 0; n + 2 < parts.length; n += 1) { + const [header, payload, signature] = [parts[n]!, parts[n + 1]!, parts[n + 2]!]; + const starts = header.indexOf(JWT_START); + if (starts !== -1 && header.length - starts >= SHORTEST_JWT_PART && payload.startsWith(JWT_START) && payload.length >= SHORTEST_JWT_PART && signature.length >= SHORTEST_JWT_SIGNATURE) { + found.push([at + starts, at + header.length + payload.length + signature.length + 2]); + at += payload.length + signature.length + 2; + n += 2; + } + at += header.length + 1; + } + return found; +} + +function jwtsIn(text: string): readonly Span[] { + const found: Span[] = []; + let runStart = -1; + for (let at = 0; at <= text.length; at += 1) { + if (at < text.length && inAJwt(text.charCodeAt(at))) { + if (runStart === -1) runStart = at; + continue; + } + if (runStart !== -1 && at - runStart >= SHORTEST_JWT) for (const jwt of jwtsInRun(text.slice(runStart, at), runStart)) found.push(jwt); + runStart = -1; + } + return found; +} + +const SCANNED_AT_ONCE = 8 * 1024 * 1024; +const LONGEST_JWT = 64 * 1024; + +export function holdsJwt(bytes: Buffer): boolean { + for (let at = 0; at < bytes.length; at += SCANNED_AT_ONCE - LONGEST_JWT) { + if (jwtsIn(bytes.toString('latin1', at, at + SCANNED_AT_ONCE)).length > 0) return true; + } + return false; +} + +export function redact(text: string): string { + let out = text; + for (const value of [...knownValues].sort((a, b) => b.length - a.length)) { + if (out.includes(value)) out = out.split(value).join(''); + } + let safe = ''; + let from = 0; + for (const [start, end] of jwtsIn(out)) { + safe += `${out.slice(from, start)}`; + from = end; + } + return safe + out.slice(from); +} + +export class Secret { + readonly name: Name; + #value: string; + constructor(name: Name, value: string) { + this.name = name; + this.#value = value; + protect(value); + } + toString(): string { + return ``; + } + toJSON(): string { + return ``; + } + [Symbol.for('nodejs.util.inspect.custom')](): string { + return ``; + } + use(_sink: SecretSink, fn: (plain: string) => T): T { + return fn(this.#value); + } +} diff --git a/integration/expo-native/specs/support/tapping.ts b/integration/expo-native/specs/support/tapping.ts new file mode 100644 index 00000000000..593c4aba222 --- /dev/null +++ b/integration/expo-native/specs/support/tapping.ts @@ -0,0 +1,21 @@ +import { NO_BUSY_WAIT, onceTheRunnerIsFree, type BusyWait } from './busy-runner.ts'; + +export const COVERED = 'covered by another visible element'; + +export interface TapSettle { + readonly timeoutMs: number; + now(): number; + wait(): Promise; +} + +export async function tapOnceUncovered(tap: () => Promise, settle: TapSettle, busy: BusyWait = NO_BUSY_WAIT): Promise { + const deadline = settle.now() + settle.timeoutMs; + for (;;) { + try { + return await onceTheRunnerIsFree(tap, busy); + } catch (error) { + if (!(error instanceof Error) || !error.message.includes(COVERED) || settle.now() >= deadline) throw error; + await settle.wait(); + } + } +} diff --git a/integration/expo-native/specs/support/types.ts b/integration/expo-native/specs/support/types.ts new file mode 100644 index 00000000000..af4fe4392f7 --- /dev/null +++ b/integration/expo-native/specs/support/types.ts @@ -0,0 +1,94 @@ +import type { Locator } from 'e2e'; + +declare const brand: unique symbol; +export type Brand = T & { readonly [brand]: B }; + +export type Platform = 'ios' | 'android'; + +export type AuthMode = 'signIn' | 'signUp' | 'signInOrUp'; + +export type RunId = Brand; +export type LaunchId = Brand; +export type StorageScope = Brand; +export type PublishableKey = Brand; +export type TestEmail = Brand; +export type TestPhone = Brand; + +export const APP_ELEMENT_IDS = { + signIn: 'e2e.auth.signIn', + signInFullScreen: 'e2e.auth.signInFullScreen', + signedOut: 'e2e.auth.signedOut', + signedIn: 'e2e.auth.signedIn', + userId: 'e2e.auth.userId', + sessionId: 'e2e.auth.sessionId', + signOut: 'e2e.auth.signOut', + error: 'e2e.launch.error', +} as const; + +export type AppLocators = { readonly [K in keyof typeof APP_ELEMENT_IDS]: Locator }; + +export const signedInText = (email: string): string => `Signed in as ${email}`; + +export const CLERK_TEST_CODE = '424242' as const; + +export interface SecretLike { + readonly name: string; + use(sink: SecretSink, fn: (plain: string) => T): T; +} + +export type SecretSink = 'bapi-authorization' | 'launch-argument' | 'platform-authorization' | 'one-password-read' | 'bapi-user-password' | 'device-input' | 'gateway-provider' | 'e2e-agent-environment' | 'test-process-environment'; + +export interface HostLaunch { + readonly verifyPublishableKey: PublishableKey; + readonly verifyRunId: RunId; + readonly verifyStorageScope: StorageScope; + readonly verifyLaunchId: LaunchId; + readonly verifyAuthMode?: AuthMode; + readonly verifyInitialIdentifier?: string; + readonly verifySignInTicket?: SecretLike; + readonly verifyLogLevel?: 'debug'; +} + +export type AppEntry = + | { readonly kind: 'binary' } + | { + readonly kind: 'dev-client'; + readonly launchArguments: readonly string[]; + readonly openLink: string | null; + readonly androidActivity: string | null; + }; + +export interface SeededUser { + readonly id: string; + readonly email: TestEmail; + readonly phone: TestPhone | null; + readonly password: SecretLike | null; +} + +export interface SeedOptions { + readonly phone?: boolean; + readonly password?: boolean; +} + +export interface LaunchOptions { + readonly authMode?: AuthMode; + readonly initialIdentifier?: string; + readonly debugLogs?: boolean; + readonly keepStorage?: boolean; + readonly signedInAs?: SeededUser; + readonly landsOn?: Locator; +} + +export interface HostFixture { + readonly runId: RunId; + readonly app: AppLocators; + newEmail(): Promise; + newPhone(): Promise; + seedUser(options?: SeedOptions): Promise; + launch(options?: LaunchOptions): Promise; + expectSignedInAs(who: SeededUser | TestEmail, timeoutMs?: number): Promise; + expectSignedOut(timeoutMs?: number): Promise; + screenshot(label: string): Promise; + tap(target: Locator): Promise; + fill(target: Locator, text: string | SecretLike): Promise; +} diff --git a/integration/expo-native/specs/support/typing.ts b/integration/expo-native/specs/support/typing.ts new file mode 100644 index 00000000000..ab1a8ec8f40 --- /dev/null +++ b/integration/expo-native/specs/support/typing.ts @@ -0,0 +1,121 @@ +import { NO_BUSY_WAIT, isRunnerBusy, type BusyWait } from './busy-runner.ts'; + +export interface FocusedField { + type(text: string): Promise; + valueIfReadable(): Promise; + valuesOfTheNamedNodes(): Promise; + refocus(): Promise; + tapAgain(): Promise; + replace(text: string): Promise; + nothingFocused?(): Promise; +} + +export const NOTHING_FOCUSED = 'TEXT_INPUT_NOT_FOCUSED'; + +const TYPED_PER_COMMAND = 16; + +export interface Settle { + readonly reads: number; + wait(): Promise; +} + +type Reading = { readonly state: 'holds' | 'unreadable' } | { readonly state: 'empty' | 'differs'; readonly held: number }; + +const lettersAndDigits = (text: string): string => text.toLowerCase().replace(/[^\p{L}\p{N}]/gu, ''); + +function compare(value: string | null, text: string): Reading { + if (value === null) return { state: 'unreadable' }; + if (value === '') return { state: 'empty', held: 0 }; + const typed = lettersAndDigits(text); + const shown = lettersAndDigits(value); + if (typed === '' || shown === '') return { state: 'unreadable' }; + return shown.includes(typed) ? { state: 'holds' } : { state: 'differs', held: shown.length }; +} + +async function read(field: FocusedField, text: string): Promise { + const focused = await field.valueIfReadable(); + if (focused !== null) return compare(focused, text); + const shown = new Set((await field.valuesOfTheNamedNodes()).filter((value) => value !== '')); + return compare(shown.size === 1 ? [...shown][0]! : null, text); +} + +async function settled(field: FocusedField, text: string, settle: Settle): Promise { + let reading = await read(field, text); + for (let reads = 1; reads < settle.reads && (reading.state === 'empty' || reading.state === 'differs'); reads += 1) { + await settle.wait(); + reading = await read(field, text); + } + return reading; +} + +export function commandsFor(text: string): readonly [string, ...string[]] { + const characters = [...text]; + const rest: string[] = []; + for (let at = TYPED_PER_COMMAND; at < characters.length; at += TYPED_PER_COMMAND) rest.push(characters.slice(at, at + TYPED_PER_COMMAND).join('')); + return [characters.slice(0, TYPED_PER_COMMAND).join(''), ...rest]; +} + +async function typeIntoAFocusedField(field: FocusedField, text: string): Promise { + if (text !== '' && (await field.nothingFocused?.()) === true) throw new Error(`${NOTHING_FOCUSED}: no text field has focus, so nothing would be typed`); + await field.type(text); +} + +async function typeOnceTheTapLanded(field: FocusedField, text: string): Promise { + const [first, ...rest] = commandsFor(text); + try { + await typeIntoAFocusedField(field, first); + } catch (error) { + if (!(error instanceof Error) || !error.message.includes(NOTHING_FOCUSED)) throw error; + await field.tapAgain(); + await typeIntoAFocusedField(field, first); + } + for (const next of rest) await field.type(next); +} + +const unlessTheRunnerWasBusy = (error: unknown): void => { + if (isRunnerBusy(error)) throw error; +}; + +async function replaceContents(field: FocusedField, text: string, busy: BusyWait): Promise { + const [first, ...rest] = commandsFor(text); + for (;;) { + try { + await field.replace(first).catch(unlessTheRunnerWasBusy); + for (const next of rest) await field.type(next).catch(unlessTheRunnerWasBusy); + return; + } catch (refusal) { + if (!(await busy.waited())) throw refusal; + } + } +} + +async function typedOrReplaced(field: FocusedField, text: string, busy: BusyWait, typing: () => Promise): Promise { + try { + await typing(); + } catch (refusal) { + if (!isRunnerBusy(refusal) || !(await busy.waited())) throw refusal; + await replaceContents(field, text, busy).catch((error: unknown) => { + throw isRunnerBusy(error) ? refusal : error; + }); + } +} + +export async function typeConfirmed(field: FocusedField, text: string, settle: Settle, busy: BusyWait = NO_BUSY_WAIT): Promise { + await typedOrReplaced(field, text, busy, () => typeOnceTheTapLanded(field, text)); + if (text === '') return; + let reading = await settled(field, text, settle); + if (reading.state === 'empty') { + await field.refocus(); + await typedOrReplaced(field, text, busy, async () => { + for (const command of commandsFor(text)) await field.type(command); + }); + reading = await settled(field, text, settle); + if (reading.state === 'empty') throw new Error('the text never reached the field: it was typed twice, and the focused field still reads empty'); + } + if (reading.state !== 'differs') return; + await replaceContents(field, text, busy); + reading = await settled(field, text, settle); + if (reading.state === 'empty' || reading.state === 'differs') { + throw new Error(`the field does not hold the typed text: ${lettersAndDigits(text).length} letters and digits were typed, and after one attempt to replace its contents the field holds ${reading.held}`); + } +} diff --git a/integration/expo-native/specs/support/waiting.ts b/integration/expo-native/specs/support/waiting.ts new file mode 100644 index 00000000000..4bddfd3bbc4 --- /dev/null +++ b/integration/expo-native/specs/support/waiting.ts @@ -0,0 +1,158 @@ +import { signedInText, type SeededUser, type TestEmail } from './types.ts'; + +export const SAVE_PASSWORD_PROMPT = 'Save Password?'; + +export const errorScreenElseSavePasswordPrompt = (errorScreenId: string): string => `id="${errorScreenId}" || label="${SAVE_PASSWORD_PROMPT}"`; + +export interface ScreenElement { + allTextContents(): Promise; +} + +export type Read = { readonly texts: readonly string[] } | { readonly unread: string }; + +export interface Reads { + of(what: string, element: ScreenElement): Promise; + summary(): string; +} + +export function timedReads(now: () => number): Reads { + let count = 0; + let slowest = { what: 'none', ms: 0 }; + let failed = 0; + let lastFailure = ''; + return { + async of(what, element) { + const startedAt = now(); + count += 1; + let read: Read; + try { + read = { texts: await element.allTextContents() }; + } catch (error) { + read = { unread: (error instanceof Error ? error.message : String(error)).replace(/\s+/g, ' ') }; + failed += 1; + lastFailure = `${what}: ${read.unread}`; + } + const ms = now() - startedAt; + if (ms > slowest.ms) slowest = { what, ms }; + return read; + }, + summary: () => `reads: ${count}, the slowest ${slowest.ms}ms (${slowest.what})${failed === 0 ? '' : `; failed reads: ${failed}, the last: ${lastFailure}`}`, + }; +} + +const only = (read: Read): string | null => ('texts' in read && read.texts.length === 1 ? read.texts[0]! : null); +const shown = (read: Read): boolean => 'texts' in read && read.texts.length > 0; +const absent = (read: Read): boolean => 'texts' in read && read.texts.length === 0; +const textsOf = (read: Read): readonly string[] => ('texts' in read ? read.texts : []); + +export interface Home { + readonly signedIn: ScreenElement; + readonly userId: ScreenElement; + readonly sessionId: ScreenElement; + readonly signedOut: ScreenElement; +} + +const HOME_ELEMENTS: { readonly [K in keyof Home]: string } = { signedIn: 'the heading', userId: 'the user ID', sessionId: 'the session ID', signedOut: '"Signed out"' }; + +export type See = (what: string, element: ScreenElement) => Promise; + +export type Sight = (see: See) => Promise; + +type ReadHome = (element: keyof Home) => Promise; + +const onTheHome = + (see: See, home: Home): ReadHome => + (element) => + see(HOME_ELEMENTS[element], home[element]); + +export const signedInAs = + (home: Home, who: SeededUser | TestEmail): Sight => + async (see) => { + const read = onTheHome(see, home); + const email = typeof who === 'string' ? who : who.email; + const isTheUser = (id: string | null): boolean => (typeof who === 'string' ? (id ?? '') !== '' : id === who.id); + return only(await read('signedIn')) === signedInText(email) && isTheUser(only(await read('userId'))) && (only(await read('sessionId')) ?? '') !== ''; + }; + +export const signedOut = + (home: Home): Sight => + async (see) => { + const read = onTheHome(see, home); + return shown(await read('signedOut')) && absent(await read('signedIn')) && absent(await read('userId')) && absent(await read('sessionId')); + }; + +export const onScreen = + (element: ScreenElement, what: string): Sight => + async (see) => + shown(await see(what, element)); + +async function homeShows(read: ReadHome): Promise { + const heading = await read('signedIn'); + const userId = await read('userId'); + const sessionId = await read('sessionId'); + const signedOutText = await read('signedOut'); + const ids = [userId, sessionId].map((id) => textsOf(id).join(' and ') || 'none'); + const parts = [ + ...textsOf(heading).map((text) => `"${text}"`), + ...(shown(signedOutText) ? ['"Signed out"'] : []), + ...(shown(heading) || shown(userId) || shown(sessionId) ? [`user ID ${ids[0]}`, `session ID ${ids[1]}`] : []), + ]; + if (parts.length > 0) return `the home shows ${parts.join(', ')}`; + return [heading, userId, sessionId, signedOutText].some((one) => 'unread' in one) ? 'the home could not be read' : 'the home is not on screen'; +} + +interface Look { + readonly reached: boolean; + readonly shows: string; +} + +async function lookAt(reads: Reads, sight: Sight, homeToDescribe: Home | null): Promise { + const taken = new Map(); + const unread: string[] = []; + const see: See = async (what, element) => { + const known = taken.get(element); + if (known !== undefined) return known; + const read = await reads.of(what, element); + taken.set(element, read); + if ('unread' in read) unread.push(`${what} (${read.unread})`); + return read; + }; + const home = homeToDescribe === null ? 'the home was not read in full' : await homeShows(onTheHome(see, homeToDescribe)); + const reached = await sight(see); + return { reached, shows: unread.length === 0 ? home : `${home}; unread: ${unread.join(', ')}` }; +} + +export interface WaitScreen { + readonly home: Home; + readonly errorScreenElsePrompt: ScreenElement; + dismissPrompt(): Promise; +} + +export interface WaitPace { + readonly timeoutMs: number; + now(): number; + wait(): Promise; +} + +export async function until(sight: Sight, waitingFor: string, screen: WaitScreen, pace: WaitPace): Promise { + const reads = timedReads(pace.now); + const deadline = pace.now() + pace.timeoutMs; + for (let pass = 1; ; pass += 1) { + const lastPass = pace.now() >= deadline; + const failure = (why: string): Error => new Error(`the app did not show ${waitingFor}${why}; passes: ${pass}; ${reads.summary()}`); + let look = await lookAt(reads, sight, lastPass ? screen.home : null); + if (look.reached) return; + const covering = await reads.of('the error screen or the save-password prompt', screen.errorScreenElsePrompt); + const inFront = textsOf(covering); + if (inFront.some((text) => text !== SAVE_PASSWORD_PROMPT)) throw failure(`; it shows its error screen: ${inFront.join(' ')}`); + if (inFront.length > 0) { + await Promise.resolve() + .then(() => screen.dismissPrompt()) + .catch(() => undefined); + look = await lookAt(reads, sight, lastPass ? screen.home : null); + if (look.reached) return; + } + if (lastPass) throw failure(` within ${pace.timeoutMs}ms; ${look.shows}`); + if (pace.now() < deadline) await pace.wait(); + } +} diff --git a/integration/expo-native/src/core/MANIFEST b/integration/expo-native/src/core/MANIFEST new file mode 100644 index 00000000000..79297d099c2 --- /dev/null +++ b/integration/expo-native/src/core/MANIFEST @@ -0,0 +1,36 @@ +0ec17d3e3a153410d40bd6cc57ddb259745c77403fa681c6670a3aa6456aa826 e2e.config.ts +38e8e99663059810b616b591dfd79bdcc7781f72b7269875294f3702b3edde84 specs/support/agent.ts +6d1ef7700a89e19d0b1214d5a6815ba757d02857c555b8bf108f2e0c9a64aad3 specs/support/busy-runner.ts +b7ecd13380d680a8a0a184cb9ef325baf97804d8b98ea7d676b21ccaf2630f4d specs/support/clerk.ts +bf7db40cc9dc4a514c3b057a34e8500e5d95130da3cbe6d0607162d3b68a28c2 specs/support/config.ts +3d77b71e24d5c3293213392dd4d9b050b9de63a26f07d362610c22bf84b8c495 specs/support/device.ts +cec7a42bf8d35e174fd3387cbcc638c01bacda74fc15ea1b14eda8abaa200d34 specs/support/fill.ts +68678a0af8e94c4498fc56321de240b097b878e9fd78adc4104c915b38a72319 specs/support/inputs.ts +18d6c5b55fc7cd71d3586466a7da4fd1e21b54a59d7c33077cc46361a0f5b959 specs/support/launch.ts +2065bc90a33659fd8f5a6e54e8b0fa57a50edc415063d4ac7e0dab744972e1b7 specs/support/secret.ts +7446b59792b957f2ace72e6c2eceb305bde25378d1f879e2884706c3d7f9f3a4 specs/support/tapping.ts +55260df9db466d21ba3a74e9ea16844f7bcda536b1e0d580b7d1ac17da257a05 specs/support/types.ts +c9b5d5ebda565670f351cf7ed0b8fe3fa589cb1cb40f8b65eca81739b3d76810 specs/support/typing.ts +10bcf31d87a05655fc263ed810a0e10c0b63a77265720d7f22f3720052f67c38 specs/support/waiting.ts +792ec6414c7e95446346abe2db850f6e4cc865a7c83a21892166fae969654277 src/core/agent.ts +1b144b8f7362be2019f578bbdf94b7e6fe3196440de5c79d4f790fa8394374e3 src/core/broker.ts +7c741449c755524bba80f1ce314554293003a34f7f7c30696015ba1c0c3dc1ab src/core/claims.ts +1fc59d13e7197e6c7099c0467744b978d498d1fe409c88254d44e2f99913a7b5 src/core/clerk.ts +15dbedb7314bf057a92cabd7900ba7533bb8ec16f6e2ede792e8ecd2b922bfa1 src/core/cli.ts +c658cd8472299371aee318f152d7a183f10e576dc0fba26904c3352f10cf5c36 src/core/devices.ts +3ae461d6aa75a2cefac0392a9ad124aec4105ac25b9320371743e2ea90fe2202 src/core/driver.ts +b59071aaefeec02dea89bd25615f8ada0da0a51e805a2523011b8b591e7435d3 src/core/e2e.ts +8c0ae6c44155105ce15011176bc630c59d6b4a2d2aa050f3fad4aa9674386ad8 src/core/evidence.ts +27bfd4ce59937aa8c97d9fd6849cf52f3da05626fbc9388385f4b5e684577d36 src/core/exec.ts +e2075f17f52ded8dea64d8a2473820396ca5ac1a22db59f21a092da28e55911f src/core/github-report.ts +39a849e9c07ce23e8a1be9f50a136690deba3cd22b153d66cc3f9f2f53d88576 src/core/keys.ts +72e7a64c6974afdc4612084d2e838075a9ce0edb6c676b23c266b8026400badc src/core/launch.d.mts +1287fc7e3328699e104c7bc50d3cd9bf9fbfec9efe764592c98e90e56033273d src/core/launch.mjs +a0ee9e4bfd34d4659685108e5e54eb1ec2aff6248236c358e6d3b14b2e2ff6a0 src/core/ledgers.ts +8d533755b21beb1e1d74d160e2740c0b1c4c4b4bc0c33f8fac75083ffeb03e75 src/core/manifest.ts +1f38ad65e58f7b4b62bb2c84bd4a5a16f5481cfb6c5f9b4b2ba059677d900bf3 src/core/publish.ts +685eabf010c7f63d076cfd3da1a681b5cc8024720538c1bf3a99792b563d8b60 src/core/slot.ts +a8838aaf1fa9e4a6bc4e350ea674079201da76c5962656039b96db2505591aed src/core/state.ts +ce8fad4756866cc6940d1fcc653640cabcc47861c34185d98211e67b9a4a5847 src/core/types.ts +5b137ac609685c490ac00b1d1ebce0ca0bb540316bfc14bb6c7a6e5b49cfa32e src/core/verbs.ts +56d2be8ec46aa0e0cf89747933e03f41fa7f3e6bed7e2a56b9d0b6e5e4db03e3 src/core/workspace.ts diff --git a/integration/expo-native/src/core/agent.ts b/integration/expo-native/src/core/agent.ts new file mode 100644 index 00000000000..1bff2e473fe --- /dev/null +++ b/integration/expo-native/src/core/agent.ts @@ -0,0 +1,41 @@ +import { UnusableAgentKey, isSet, readAgent, type Agent } from '../../specs/support/agent.ts'; +import { AGENT_CREDENTIAL_VARIABLES } from './launch.mjs'; +import { VerifyFailure, type DoctorCheck } from './types.ts'; + +export type AgentSource = () => Agent | null; + +type Env = Readonly>; + +function readOrKeysMissing(env: Env): Agent | null { + try { + return readAgent(env); + } catch (error) { + if (!(error instanceof UnusableAgentKey)) throw error; + throw new VerifyFailure('KEYS_MISSING', error.what, error.fix); + } +} + +export function takeAgent(env: NodeJS.ProcessEnv): AgentSource { + const taken = { AI_GATEWAY_API_KEY: env.AI_GATEWAY_API_KEY, AI_GATEWAY_API_KEY_FILE: env.AI_GATEWAY_API_KEY_FILE }; + for (const name of AGENT_CREDENTIAL_VARIABLES) delete env[name]; + let read: { readonly agent: Agent | null } | null = isSet(taken.AI_GATEWAY_API_KEY) ? { agent: readOrKeysMissing(taken) } : null; + return () => (read ??= { agent: readOrKeysMissing(taken) }).agent; +} + +export function agentEnvironment(agent: Agent | null): Readonly> { + if (agent === null) return {}; + const found = agent.credential; + return found.variable === 'AI_GATEWAY_API_KEY_FILE' ? { AI_GATEWAY_API_KEY_FILE: found.file } : { AI_GATEWAY_API_KEY: found.key.use('e2e-agent-environment', (plain) => plain) }; +} + +export function agentCheck(source: AgentSource): DoctorCheck { + let agent: Agent | null; + try { + agent = source(); + } catch (error) { + if (!(error instanceof VerifyFailure)) throw error; + return { id: 'agent', ok: false, detail: error.message, fix: error.fix }; + } + if (agent === null) return { id: 'agent', ok: true, detail: 'none: AI_GATEWAY_API_KEY and AI_GATEWAY_API_KEY_FILE are not set, so agent.act and agent.assert have no model' }; + return { id: 'agent', ok: true, detail: `${agent.model}, with ${agent.backup} as its backup, through the Vercel AI Gateway; key from ${agent.credential.variable}` }; +} diff --git a/integration/expo-native/src/core/broker.ts b/integration/expo-native/src/core/broker.ts new file mode 100644 index 00000000000..46932884a1b --- /dev/null +++ b/integration/expo-native/src/core/broker.ts @@ -0,0 +1,197 @@ +import { randomBytes, timingSafeEqual } from 'node:crypto'; +import { writeFileSync } from 'node:fs'; +import { createServer, type IncomingMessage, type ServerResponse } from 'node:http'; +import { join } from 'node:path'; +import { BACKEND_API_URL, BACKEND_CALLS, TICKET_SECONDS, parseTestEmail, parseTestPhone, runEmailPrefix, typedPassword, type BackendCall } from '../../specs/support/clerk.ts'; +import { protect } from '../../specs/support/secret.ts'; +import { ownKeyRefused, userLimitReached } from './clerk.ts'; +import type { InstanceKeys } from './keys.ts'; +import { newEntryId, type Workspace } from './workspace.ts'; +import { VerifyFailure, type RunId, type ScratchPath, type TestEmail } from './types.ts'; + +export const STAND_IN_PATH = '/v1'; + +const BODY_LIMIT_BYTES = 64 * 1024; + +export interface BrokerDeps { + readonly keys: () => InstanceKeys; + readonly fetch: typeof fetch; +} + +export interface Broker { + readonly url: string; + readonly tokenFile: string; + stop(): Promise; +} + +interface Asked { + readonly query: URLSearchParams; + readonly body: Readonly>; +} + +interface Guard { + refuses(asked: Asked): string | null; + forwards(asked: Asked): Asked; + answered?(asked: Asked, reply: unknown): unknown; +} + +const NO_QUERY = new URLSearchParams(); + +interface Answer { + readonly status: number; + readonly body: string; +} + +const clerkError = (status: number, code: string, message: string): Answer => ({ status, body: JSON.stringify({ errors: [{ code, message, long_message: message }] }) }); + +const refused = (why: string): Answer => clerkError(403, 'verify_stand_in_refused', `the verify CLI's stand-in for the Backend API refused this request: ${why}`); + +const failure = (status: number, code: string, error: VerifyFailure): Answer => clerkError(status, code, `${error.message} (fix: ${error.fix})`); + +const onlyKeys = (given: Readonly> | URLSearchParams, allowed: readonly string[]): string | null => { + const extra = [...(given instanceof URLSearchParams ? given.keys() : Object.keys(given))].filter((key) => !allowed.includes(key)); + return extra.length === 0 ? null : `it carries ${extra.join(', ')}, and the tests send only ${allowed.join(', ')}`; +}; + +const notOurs = (check: () => unknown): string | null => { + try { + check(); + return null; + } catch (error) { + return (error as Error).message; + } +}; + +export async function startBroker(run: RunId, workspace: Workspace, scratch: ScratchPath, deps: BrokerDeps): Promise { + const token = randomBytes(32).toString('hex'); + protect(token); + const tokenFile = join(scratch, 'broker-token'); + writeFileSync(tokenFile, token, { mode: 0o600 }); + const created = new Set(); + const prefix = runEmailPrefix(run); + + const guards: Record = { + createUser: { + refuses({ body }) { + const emails = body.email_address; + if (!Array.isArray(emails) || emails.length !== 1 || typeof emails[0] !== 'string') return 'a user is created with exactly one email address'; + const email: string = emails[0]; + const phones = body.phone_number === undefined ? [] : body.phone_number; + if (!Array.isArray(phones) || phones.some((phone) => typeof phone !== 'string')) return 'phone_number is not a list of phone numbers'; + return ( + onlyKeys(body, ['email_address', 'phone_number', 'password', 'skip_password_requirement', 'bypass_client_trust']) ?? + notOurs(() => parseTestEmail(email)) ?? + (email.startsWith(prefix) ? null : `${email} is not an address of run ${run}, whose addresses start ${prefix}`) ?? + phones.map((phone: string) => notOurs(() => parseTestPhone(phone))).find((problem) => problem !== null) ?? + null + ); + }, + forwards: ({ body }) => ({ query: NO_QUERY, body: body.phone_number === undefined ? body : { ...body, phone_number: (body.phone_number as string[]).map(parseTestPhone) } }), + answered({ body }, reply) { + if (typeof body.password === 'string') typedPassword(body.password); + const userId = (reply as { id?: unknown } | null)?.id; + if (typeof userId === 'string') { + created.add(userId); + workspace.append({ id: newEntryId(), kind: 'user', run, userId, email: (body.email_address as [TestEmail])[0] }); + } + return reply; + }, + }, + signInToken: { + refuses({ body }) { + const { user_id: userId, expires_in_seconds: seconds } = body; + if (typeof userId !== 'string' || !created.has(userId)) return `user ${String(userId)} was not created by this run; sign in only users from host.seedUser`; + if (typeof seconds !== 'number' || !(seconds > 0 && seconds <= TICKET_SECONDS)) return `a sign-in ticket lives ${TICKET_SECONDS} seconds at most`; + return onlyKeys(body, ['user_id', 'expires_in_seconds']); + }, + forwards: ({ body }) => ({ query: NO_QUERY, body }), + answered(_asked, reply) { + const ticket = (reply as { token?: unknown } | null)?.token; + if (typeof ticket === 'string') protect(ticket); + return reply; + }, + }, + usersByPhone: { + refuses({ query }) { + const phones = query.getAll('phone_number'); + if (phones.length !== 1) return 'users are looked up by one test phone'; + return onlyKeys(query, ['phone_number']) ?? notOurs(() => parseTestPhone(phones[0]!)); + }, + forwards: ({ query }) => ({ query: new URLSearchParams({ phone_number: parseTestPhone(query.get('phone_number')!) }), body: {} }), + answered: (_asked, reply) => (Array.isArray(reply) ? reply.flatMap((user: { id?: unknown } | null) => (typeof user?.id === 'string' ? [{ id: user.id }] : [])) : reply), + }, + }; + + const guardFor = (method: string | undefined, path: string): Guard | undefined => { + const call = (Object.keys(BACKEND_CALLS) as BackendCall[]).find((name) => BACKEND_CALLS[name].method === method && `${STAND_IN_PATH}${BACKEND_CALLS[name].path}` === path); + return call === undefined ? undefined : guards[call]; + }; + + async function forward(method: string, path: string, { query, body }: Asked): Promise { + const response = await deps.keys().sk.use('bapi-authorization', (plain) => + deps.fetch(`${BACKEND_API_URL}${path.slice(STAND_IN_PATH.length)}${query.size === 0 ? '' : `?${query}`}`, { + method, + headers: { Authorization: `Bearer ${plain}`, 'Content-Type': 'application/json' }, + ...(method === 'GET' ? {} : { body: JSON.stringify(body) }), + signal: AbortSignal.timeout(30_000), + }), + ); + return { status: response.status, body: await response.text() }; + } + + async function answer(request: IncomingMessage): Promise { + const url = new URL(request.url ?? '/', 'http://127.0.0.1'); + const guard = guardFor(request.method, url.pathname); + if (guard === undefined) return clerkError(404, 'verify_stand_in_unknown_call', `the verify CLI's stand-in for the Backend API has no ${request.method} ${url.pathname}; the tests make ${Object.values(BACKEND_CALLS).map((call) => `${call.method} ${call.path}`).join(', ')}`); + let text = ''; + for await (const chunk of request) { + text += chunk; + if (text.length > BODY_LIMIT_BYTES) return refused('its body is larger than any request the tests make'); + } + let body: Record; + try { + const parsed: unknown = text.length > 0 ? JSON.parse(text) : {}; + if (typeof parsed !== 'object' || parsed === null || Array.isArray(parsed)) return refused('its body is not a JSON object'); + body = parsed as Record; + } catch { + return refused('its body is not JSON'); + } + const asked: Asked = { query: url.searchParams, body }; + const why = guard.refuses(asked); + if (why !== null) return refused(why); + const reply = await forward(request.method ?? 'GET', url.pathname, guard.forwards(asked)); + if (reply.status === 401) return failure(401, 'verify_stand_in_key_refused', ownKeyRefused()); + let json: unknown; + try { + json = reply.body.length > 0 ? JSON.parse(reply.body) : null; + } catch { + return reply; + } + if (reply.status < 200 || reply.status >= 300) { + const codes = ((json as { errors?: { code?: string }[] } | null)?.errors ?? []).map((error) => error.code); + return reply.status === 403 && codes.includes('user_quota_exceeded') ? failure(403, 'user_quota_exceeded', userLimitReached()) : reply; + } + return { status: reply.status, body: JSON.stringify(guard.answered?.(asked, json) ?? json) }; + } + + async function handle(request: IncomingMessage, response: ServerResponse): Promise { + const auth = Buffer.from(request.headers.authorization ?? ''); + const expected = Buffer.from(`Bearer ${token}`); + if (auth.length !== expected.length || !timingSafeEqual(auth, expected)) { + response.writeHead(401).end(); + return; + } + const sent = await answer(request).catch((error: unknown) => clerkError(502, 'verify_stand_in_failed', `the verify CLI's stand-in for the Backend API could not forward this request: ${(error as Error).message ?? String(error)}`)); + response.writeHead(sent.status, { 'Content-Type': 'application/json' }).end(sent.body); + } + + const server = createServer((request, response) => void handle(request, response)); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + const address = server.address(); + if (address === null || typeof address === 'string') throw new Error('broker did not bind a TCP port'); + return { + url: `http://127.0.0.1:${address.port}${STAND_IN_PATH}`, + tokenFile, + stop: () => new Promise((resolve) => server.close(() => resolve())), + }; +} diff --git a/integration/expo-native/src/core/claims.ts b/integration/expo-native/src/core/claims.ts new file mode 100644 index 00000000000..390bbb9b7b7 --- /dev/null +++ b/integration/expo-native/src/core/claims.ts @@ -0,0 +1,63 @@ +import { randomUUID } from 'node:crypto'; +import { existsSync, readdirSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { join } from 'node:path'; +import { currentProcess, isRunning, type ProcessRef } from './exec.ts'; +import { compareAndSwapSlot, readSlot } from './slot.ts'; +import type { DeviceName, Platform } from './types.ts'; + +export interface Claim { + readonly platform: Platform; + readonly slot: number; + readonly gen: number; + readonly nonce: string; + readonly deviceName: DeviceName; + readonly worktree: string; + readonly owner: ProcessRef; + readonly reaping: boolean; + readonly createdAt: string; +} + +export const defaultClaimsDir = (): string => join(homedir(), '.verify', 'claims'); + +const slotDir = (dir: string, platform: Platform, slot: number) => join(dir, `${platform}-${slot}`); + +export function readClaim(dir: string, platform: Platform, slot: number): { readonly gen: number; readonly claim: Claim | null } { + const state = readSlot(slotDir(dir, platform, slot)); + return { gen: state.gen, claim: state.value === null ? null : { ...(JSON.parse(state.value) as Omit), gen: state.gen } }; +} + +export function readClaims(dir: string, platform: Platform): readonly Claim[] { + if (!existsSync(dir)) return []; + const slots = readdirSync(dir).flatMap((name) => { + const match = new RegExp(`^${platform}-(\\d+)$`).exec(name); + return match === null ? [] : [Number(match[1])]; + }); + return slots.flatMap((slot) => { + const { claim } = readClaim(dir, platform, slot); + return claim === null ? [] : [claim]; + }); +} + +export function takeSlot(dir: string, platform: Platform, slot: number, from: number, worktree: string, reaping = false): Claim | null { + const claim: Omit = { + platform, + slot, + nonce: randomUUID(), + deviceName: `verify-${platform}-${slot}`, + worktree, + owner: currentProcess(), + reaping, + createdAt: new Date().toISOString(), + }; + return compareAndSwapSlot(slotDir(dir, platform, slot), from, JSON.stringify(claim)) ? { ...claim, gen: from + 1 } : null; +} + +export function freeSlot(dir: string, claim: Claim): boolean { + return compareAndSwapSlot(slotDir(dir, claim.platform, claim.slot), claim.gen, null); +} + +export function isOrphaned(claim: Claim): boolean { + if (claim.reaping) return !isRunning(claim.owner); + return !existsSync(claim.worktree) && !isRunning(claim.owner); +} diff --git a/integration/expo-native/src/core/clerk.ts b/integration/expo-native/src/core/clerk.ts new file mode 100644 index 00000000000..76ab99cd36f --- /dev/null +++ b/integration/expo-native/src/core/clerk.ts @@ -0,0 +1,82 @@ +import { runEmailPrefix } from '../../specs/support/clerk.ts'; +import type { InstanceKeys } from './keys.ts'; +import { VerifyFailure, type PublishableKey, type RunId, type TestEmail } from './types.ts'; + +export function frontendApiHost(pk: PublishableKey): string { + const decoded = Buffer.from(pk.replace(/^pk_(test|live)_/, ''), 'base64url').toString('utf8'); + return decoded.replace(/\$$/, ''); +} + +export interface RunUser { + readonly userId: string; + readonly email: TestEmail; +} + +export interface ClerkBackend { + usersOfRun(run: RunId): Promise; + userCount(): Promise; +} + +export const DEVELOPMENT_USER_LIMIT = 100; +export const REPLACE_AT_USERS = 60; +export const USERS_PAGE_LIMIT = 500; + +export const BACKEND_API_HOST = 'api.clerk.com'; + +export const ownKeyRefused = (): VerifyFailure => + new VerifyFailure( + 'NOT_READY', + `${BACKEND_API_HOST} answered 401 to the instance's own secret key; the likely cause is a cloud environment whose API credential for ${BACKEND_API_HOST} has no path prefix, so it replaces the key on every request to that host`, + `set Path prefixes on that credential to /v1/platform/, so it is attached to Platform API calls only, then rerun`, + ); + +export const userLimitReached = (): VerifyFailure => + new VerifyFailure('INSTANCE_MISCONFIGURED', `the instance holds the ${DEVELOPMENT_USER_LIMIT} users a development instance allows, and Clerk refused one more`, `the next \`{cli} run\` replaces the instance once it holds ${REPLACE_AT_USERS} users; rerun`); + +class ClerkHttpError extends Error { + constructor(status: number, codes: readonly string[], path: string) { + super(`Clerk ${path} answered ${status}${codes.length ? ` (${codes.join(', ')})` : ''}`); + } +} + +export function createClerkBackends(fetchImpl: typeof fetch = fetch): (keys: () => InstanceKeys) => ClerkBackend { + async function request(keys: InstanceKeys, method: string, path: string): Promise { + const response = await keys.sk.use('bapi-authorization', (plain) => + fetchImpl(`https://${BACKEND_API_HOST}/v1${path}`, { + method, + headers: { Authorization: `Bearer ${plain}`, 'Content-Type': 'application/json' }, + signal: AbortSignal.timeout(30_000), + }), + ); + const text = await response.text(); + if (response.status === 401) throw ownKeyRefused(); + let json: unknown = null; + try { + json = text.length > 0 ? JSON.parse(text) : null; + } catch { + json = null; + } + if (response.status < 200 || response.status >= 300) { + const errors = (json as { errors?: { code?: string }[] } | null)?.errors ?? []; + throw new ClerkHttpError(response.status, errors.map((e) => e.code ?? 'unknown'), `${method} ${path.split('?')[0]}`); + } + return json; + } + + return (keys) => ({ + async usersOfRun(run) { + const prefix = runEmailPrefix(run); + const listed = await request(keys(), 'GET', `/users?limit=${USERS_PAGE_LIMIT}&email_address_query=${encodeURIComponent(prefix)}`); + if (!Array.isArray(listed)) return []; + return listed.flatMap((user: { id?: unknown; email_addresses?: { email_address?: unknown }[] } | null) => { + const email = (user?.email_addresses ?? []).map((address) => address.email_address).find((address): address is string => typeof address === 'string' && address.startsWith(prefix)); + return typeof user?.id === 'string' && email !== undefined ? [{ userId: user.id, email: email as TestEmail }] : []; + }); + }, + async userCount() { + const counted = (await request(keys(), 'GET', '/users/count')) as { total_count?: unknown } | null; + if (typeof counted?.total_count !== 'number') throw new Error('Clerk counted the users of the instance without a total'); + return counted.total_count; + }, + }); +} diff --git a/integration/expo-native/src/core/cli.ts b/integration/expo-native/src/core/cli.ts new file mode 100644 index 00000000000..6c7fa99fe26 --- /dev/null +++ b/integration/expo-native/src/core/cli.ts @@ -0,0 +1,352 @@ +import { execFileSync } from 'node:child_process'; +import { basename, dirname, isAbsolute, join, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { run as defaultRunner } from './exec.ts'; +import { takeAgent } from './agent.ts'; +import { PLATFORM_CREDENTIAL_VARIABLES } from './launch.mjs'; +import { redact } from '../../specs/support/secret.ts'; +import { leaseLine } from './devices.ts'; +import { count } from './state.ts'; +import { createInstances } from './instances/instances.ts'; +import { verbs, type Deps } from './verbs.ts'; +import { openWorkspace, parseRunId } from './workspace.ts'; +import { + CLI_PLACEHOLDER, + RETRYABLE, + VerifyFailure, + type BackendKind, + type Command, + type DoctorCheck, + type HostAdapter, + type Invocation, + type Platform, + type RunResult, + type Verb, + type VerbResult, +} from './types.ts'; + +const VERBS: readonly Verb[] = ['doctor', 'up', 'run', 'screen', 'attach', 'down']; + +const USAGE_FIX = [ + '{cli} doctor [--platform p] [--backend auto|local] [--live]', + '{cli} up [--platform p] [--backend auto|local] [--wait ]', + '{cli} run ... | --all [--platform p] [--backend auto|local] [--grep re] [--retries ] [--github-report] [--no-video] [--wait ]', + '{cli} screen [--platform p] [--png]', + '{cli} attach --pr [--screenshot label]...', + '{cli} down [--platform p] [--stale] [--dry-run]', + 'every verb takes --json', +].join('; '); + +const usage = (message: string) => new VerifyFailure('USAGE', message, USAGE_FIX); + +type FlagSpec = Readonly>; + +const FLAGS: Readonly> = { + doctor: { platform: 'value', backend: 'value', live: 'bool' }, + up: { platform: 'value', backend: 'value', wait: 'value' }, + run: { platform: 'value', backend: 'value', all: 'bool', grep: 'value', retries: 'value', 'github-report': 'bool', 'no-video': 'bool', wait: 'value' }, + screen: { platform: 'value', png: 'bool' }, + attach: { pr: 'value', screenshot: 'list' }, + down: { platform: 'value', stale: 'bool', 'dry-run': 'bool' }, +}; + +function platformFlag(value: string | undefined): Platform | undefined { + if (value === undefined) return undefined; + if (value === 'ios' || value === 'android') return value; + throw usage(`--platform must be ios or android, not ${value}`); +} + +function backendFlag(value: string | undefined): BackendKind | undefined { + if (value === undefined || value === 'auto') return undefined; + if (value === 'local') return value; + throw usage(`--backend must be auto or local, not ${value}`); +} + +function positiveInt(flag: string, value: string | undefined, fallback: number | undefined): number { + if (value === undefined) { + if (fallback === undefined) throw usage(`--${flag} is required`); + return fallback; + } + if (!/^\d+$/.test(value)) throw usage(`--${flag} must be a whole number, not ${value}`); + return Number(value); +} + +export function parseArgv(argv: readonly string[]): Invocation { + const [verbArg, ...rest] = argv; + const verb = VERBS.find((v) => v === verbArg); + if (verb === undefined) throw usage(verbArg === undefined ? 'no verb given' : `unknown verb ${verbArg}`); + const spec = FLAGS[verb]; + const values = new Map(); + const lists = new Map(); + const bools = new Set(); + const positionals: string[] = []; + let json = false; + for (let i = 0; i < rest.length; i += 1) { + const arg = rest[i]!; + if (!arg.startsWith('--')) { + positionals.push(arg); + continue; + } + const [name, inline] = arg.slice(2).split(/=(.*)/s, 2) as [string, string | undefined]; + if (name === 'json' && inline === undefined) { + json = true; + continue; + } + const kind = spec[name]; + if (kind === undefined) throw usage(`{cli} ${verb} does not take --${name}`); + if (kind === 'bool') { + if (inline !== undefined) throw usage(`--${name} takes no value`); + bools.add(name); + continue; + } + const value = inline ?? rest[(i += 1)]; + if (value === undefined || value.startsWith('--')) throw usage(`--${name} needs a value`); + if (kind === 'list') lists.set(name, [...(lists.get(name) ?? []), value]); + else { + if (values.has(name)) throw usage(`--${name} given twice`); + values.set(name, value); + } + } + const noPositionals = () => { + if (positionals.length > 0) throw usage(`{cli} ${verb} takes no arguments, got ${positionals.join(' ')}`); + }; + const platform = platformFlag(values.get('platform')); + const backend = backendFlag(values.get('backend')); + const base = { ...(platform === undefined ? {} : { platform }), ...(backend === undefined ? {} : { backend }) }; + + let command: Command; + switch (verb) { + case 'doctor': + noPositionals(); + command = { verb, ...base, live: bools.has('live') }; + break; + case 'up': + noPositionals(); + command = { verb, ...base, waitSeconds: positiveInt('wait', values.get('wait'), 0) }; + break; + case 'run': { + const all = bools.has('all'); + if (all && positionals.length > 0) throw usage('pass selectors or --all, not both'); + if (!all && positionals.length === 0) throw usage('{cli} run needs a feature, feature/spec, path.e2e.ts, or --all'); + const grep = values.get('grep'); + const retries = positiveInt('retries', values.get('retries'), 0); + command = { + verb, + ...base, + selection: all ? { all: true } : { selectors: positionals }, + ...(grep === undefined ? {} : { grep }), + video: !bools.has('no-video'), + retries, + githubReport: bools.has('github-report'), + waitSeconds: positiveInt('wait', values.get('wait'), 0), + }; + break; + } + case 'screen': + noPositionals(); + command = { verb, ...(platform === undefined ? {} : { platform }), png: bools.has('png') }; + break; + case 'attach': { + if (positionals.length !== 1) throw usage('{cli} attach takes exactly one run id'); + const shots = lists.get('screenshot'); + command = { verb, run: parseRunId(positionals[0]!), pr: positiveInt('pr', values.get('pr'), undefined), screenshots: shots ?? 'all' }; + break; + } + case 'down': + noPositionals(); + command = { verb, ...(platform === undefined ? {} : { platform }), stale: bools.has('stale'), dryRun: bools.has('dry-run') }; + break; + default: { + const exhaustive: never = verb; + throw usage(`unknown verb ${String(exhaustive)}`); + } + } + return { command, json }; +} + +export interface Output { + result(value: VerbResult): void; + failure(error: VerifyFailure): void; + progress(line: string): void; +} + +interface Sink { + write(text: string): unknown; +} + +const pad = (text: string, width: number) => text.padEnd(width); + +function rel(packageDir: string, path: string): string { + return relative(packageDir, path) || path; +} + +function renderRun(result: RunResult, packageDir: string): string[] { + const r = result.record; + const lines: string[] = []; + const width = Math.max(0, ...r.results.map((x) => x.spec.path.replace(/^specs\/(golden\/)?/, '').length)); + for (const x of r.results) { + const label = { passed: 'pass', failed: 'FAIL', skipped: 'skip', flaky: 'flaky', interrupted: 'INTR' }[x.status]; + const name = x.spec.path.replace(/^specs\/(golden\/)?/, ''); + const tail = x.status === 'skipped' ? (x.skipReason ?? '') : `${x.seconds}s`; + lines.push(` ${pad(label, 5)} ${pad(name, width)} ${x.title} ${tail}`); + if (x.error !== null) lines.push(` ${x.status === 'flaky' ? `passed on attempt ${x.attempts}; the attempt before it failed: ` : ''}${x.error}`); + if (x.failurePage !== null) lines.push(` failure page ${rel(packageDir, x.failurePage)}`); + if (x.failureScreenshot !== null) lines.push(` screenshot ${rel(packageDir, x.failureScreenshot)}`); + } + const flaky = r.results.filter((x) => x.status === 'flaky').length; + if (flaky > 0) lines.push(`flaky ${count(flaky, 'test')} passed only on a retry, and the run does not fail for ${flaky === 1 ? 'it' : 'them'}`); + lines.push(`evidence ${rel(process.cwd(), result.dir)}`); + if (r.videos.length > 0) lines.push(` video ${r.videos.map((v) => basename(v)).join(', ')}`); + if (r.screenshots.length > 0) lines.push(` screenshots ${r.screenshots.map((s) => basename(s.path)).join(', ')}`); + if (r.appLog !== null) lines.push(` app log ${basename(r.appLog)}`); + if (r.tainted.length > 0) lines.push(` TAINTED ${r.tainted.map((t) => rel(result.dir, t)).join(', ')} (attach is blocked)`); + lines.push(`next ${isAbsolute(result.next) ? rel(process.cwd(), result.next) : result.next}`); + return lines; +} + +function render(value: VerbResult, packageDir: string): string[] { + switch (value.verb) { + case 'doctor': { + const width = Math.max(...value.checks.map((c) => c.id.length)); + const label = (c: DoctorCheck) => (!c.ok ? 'FAIL' : c.state === 'warning' ? 'warn' : c.state === 'not-run' ? 'skip' : 'ok'); + return value.checks.flatMap((c) => [`${pad(label(c), 5)} ${pad(c.id, width)} ${c.detail}`, ...(c.fix === undefined ? [] : [` fix: ${c.fix}`])]); + } + case 'up': + return value.leases.map(leaseLine); + case 'run': + return renderRun(value, packageDir); + case 'screen': { + const lines = [`screen ${value.platform} ${value.device}`]; + for (const node of value.nodes) { + const label = node.name ?? node.text; + if (label === null && node.testId === null) continue; + lines.push( + `${' '.repeat(Math.min(node.depth, 8))}${pad(node.role, 10)} ${label === null ? '' : JSON.stringify(label)}${node.testId === null ? '' : ` id=${node.testId}`}${node.locator === null ? '' : ` ${node.locator}`}`, + ); + } + if (value.png !== null) lines.push(`png ${rel(process.cwd(), value.png)}`); + return lines; + } + case 'attach': + return [`${value.alreadyPosted ? 'already posted' : 'posted'} ${value.posted.map((p) => basename(p)).join(', ')} ${value.commentUrl}`]; + case 'down': + return [ + ...(value.dryRun + ? [ + 'dry run: nothing was changed', + `would release ${value.wouldRelease.map((l) => l.device).join(', ') || 'nothing'}`, + `would delete ${count(value.wouldDelete.length, 'application')}`, + ...value.wouldDelete.map((target) => ` application ${target.name} (with every test user in it)`), + stoppedLine('would stop ', value.wouldStop), + ] + : [ + `released ${value.released.map((l) => l.device).join(', ') || 'nothing'}`, + `deleted ${count(value.deletedApplications.length, 'application')}${value.deletedApplications.length === 0 ? '' : ` (${value.deletedApplications.map((a) => a.name).join(', ')}, with every test user in ${value.deletedApplications.length === 1 ? 'it' : 'them'})`}`, + stoppedLine('stopped ', value.stoppedProcesses), + ]), + `kept ${count(value.keptRuns.length, 'run')} in .verify/runs/`, + ]; + default: { + const exhaustive: never = value; + return [JSON.stringify(exhaustive)]; + } + } +} + +export function createOutput(json: boolean, packageDir: string, cli: string, stdout: Sink = process.stdout, stderr: Sink = process.stderr): Output { + const text = (value: string) => redact(value).replaceAll(CLI_PLACEHOLDER, cli); + return { + result(value) { + if (json) stdout.write(`${text(JSON.stringify({ ok: true, ...value }))}\n`); + else stdout.write(`${text(render(value, packageDir).join('\n'))}\n`); + }, + failure(error) { + const body = { code: error.code, message: error.message, fix: error.fix, retryable: RETRYABLE.has(error.code) }; + if (json) stdout.write(`${text(JSON.stringify({ ok: false, error: body }))}\n`); + else stderr.write(`${text(`error ${error.code} ${error.message}\n fix: ${error.fix}`)}\n`); + }, + progress(line) { + if (!json) stderr.write(`${text(line)}\n`); + }, + }; +} + +function stoppedLine(label: string, processes: readonly string[]): string { + const daemon = processes.some((p) => p.startsWith('agent-device ')) ? '' : '; no agent-device daemon running'; + return `${label}${processes.join(', ') || 'nothing'}${daemon}`; +} + +export function exitCodeFor(value: VerbResult): number { + if (value.verb === 'doctor') return value.ok ? 0 : 3; + if (value.verb === 'run') return value.record.results.some((r) => r.status === 'failed' || r.status === 'interrupted') ? 1 : 0; + return 0; +} + +const PACKAGE_DIR = join(dirname(fileURLToPath(import.meta.url)), '..', '..'); + +export function takePlatformKey(env: NodeJS.ProcessEnv): Readonly> { + const withKeys = { ...env }; + for (const name of PLATFORM_CREDENTIAL_VARIABLES) delete env[name]; + return withKeys; +} + +export async function main(argv: readonly string[], host: HostAdapter): Promise { + let invocation: Invocation; + try { + invocation = parseArgv(argv); + } catch (error) { + const failure = error instanceof VerifyFailure ? error : usage(String(error)); + createOutput(argv.includes('--json'), PACKAGE_DIR, host.cli).failure(failure); + return 2; + } + const out = createOutput(invocation.json, PACKAGE_DIR, host.cli); + try { + const agent = takeAgent(process.env); + const withPlatformKey = takePlatformKey(process.env); + const worktree = execFileSync('git', ['rev-parse', '--show-toplevel'], { cwd: PACKAGE_DIR, encoding: 'utf8' }).trim(); + const workspace = openWorkspace({ packageDir: PACKAGE_DIR, worktree }); + const progress = (line: string) => out.progress(line); + const deps: Deps = { + host, + workspace, + runner: defaultRunner, + env: process.env, + progress, + instances: createInstances({ workspace, env: withPlatformKey, runner: defaultRunner, progress }), + agent, + }; + const command = invocation.command; + let result: VerbResult; + switch (command.verb) { + case 'doctor': + result = await verbs.doctor(deps, command); + break; + case 'up': + result = await verbs.up(deps, command); + break; + case 'run': + result = await verbs.run(deps, command); + break; + case 'screen': + result = await verbs.screen(deps, command); + break; + case 'attach': + result = await verbs.attach(deps, command); + break; + case 'down': + result = await verbs.down(deps, command); + break; + default: { + const exhaustive: never = command; + throw usage(`unknown verb ${JSON.stringify(exhaustive)}`); + } + } + out.result(result); + return exitCodeFor(result); + } catch (error) { + const failure = + error instanceof VerifyFailure ? error : new VerifyFailure('NOT_READY', (error as Error).message ?? String(error), 'run `{cli} doctor`, then retry'); + out.failure(failure); + return failure.code === 'USAGE' ? 2 : 3; + } +} diff --git a/integration/expo-native/src/core/devices.ts b/integration/expo-native/src/core/devices.ts new file mode 100644 index 00000000000..effb62ff435 --- /dev/null +++ b/integration/expo-native/src/core/devices.ts @@ -0,0 +1,210 @@ +import { createHash } from 'node:crypto'; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { join, resolve } from 'node:path'; +import type { Device } from '../../specs/support/inputs.ts'; +import type { Instances } from './instances/instances.ts'; +import { run } from './exec.ts'; +import { finishOrphanLedgers } from './ledgers.ts'; +import { newEntryId, type Workspace } from './workspace.ts'; +import { + VerifyFailure, + type AcquireLock, + type Availability, + type BackendKind, + type BuildKey, + type BuildView, + type BuiltApp, + type DeviceBackend, + type HostAdapter, + type Lease, + type LeaseView, + type LocalBuild, + type ProcessRef, + type Platform, + type ScratchPath, +} from './types.ts'; + +export interface LeaseOutcome { + readonly lease: Lease; + readonly backend: DeviceBackend; + readonly app: BuiltApp; + readonly view: LeaseView; + readonly build: BuildView; +} + +export const deviceOf = (lease: Lease): Device => ({ kind: 'local', id: lease.deviceId }); + +export function backendFor(host: HostAdapter, platform: Platform, kind: BackendKind): DeviceBackend { + const backend = host.backends.find((b) => b.platform === platform && b.kind === kind); + if (backend === undefined) { + const others = host.backends.filter((b) => b.platform === platform); + const fix = others.length === 0 ? `${host.repo} has no ${platform} backend` : `use ${others.map((b) => `--backend ${b.kind} (needs ${b.requirement})`).join(' or ')}`; + throw new VerifyFailure('UNSUPPORTED', `${host.repo} has no ${kind} backend for ${platform}`, fix); + } + return backend; +} + +export interface BackendChoice { + readonly backend: DeviceBackend; + readonly why: string; +} + +const toUseIt = (availability: Availability): string => (availability.fix === undefined ? '' : ` (to run it here: ${availability.fix})`); + +export function selectBackend(host: HostAdapter, platform: Platform, requested: BackendKind | undefined, held: Lease | null): BackendChoice { + if (requested !== undefined) { + const backend = backendFor(host, platform, requested); + const forced = backend.availability(); + if (!forced.usable) throw new VerifyFailure('UNSUPPORTED', `--backend ${requested} cannot run here: ${forced.why}`, forced.fix ?? `drop --backend so the CLI picks one, or run on ${backend.requirement}`); + return { backend, why: `forced by --backend ${requested}` }; + } + if (held !== null) return { backend: backendFor(host, platform, held.backend), why: `this worktree already holds a ${held.backend} lease` }; + const candidates = host.backends.filter((b) => b.platform === platform).map((backend) => ({ backend, availability: backend.availability() })); + const chosen = candidates.find((c) => c.availability.usable); + if (chosen === undefined) { + const out = candidates.map((c) => `${c.backend.kind}: ${c.availability.why}${toUseIt(c.availability)}`).join('; '); + throw new VerifyFailure('UNSUPPORTED', `no ${platform} backend runs on this machine${out === '' ? '' : ` (${out})`}`, candidates.length === 0 ? `${host.repo} has no ${platform} backend` : `run on ${candidates.map((c) => c.backend.requirement).join(' or ')}`); + } + const passed = candidates.slice(0, candidates.indexOf(chosen)).map((c) => `${c.backend.kind} is out: ${c.availability.why}${toUseIt(c.availability)}`); + return { backend: chosen.backend, why: [...passed, chosen.availability.why].join('; ') }; +} + +export const describeChoice = (choice: BackendChoice): string => `${choice.backend.kind} ${choice.why}`; + +export function leaseView(backend: DeviceBackend, lease: Lease, renewed: boolean): LeaseView { + return { + platform: lease.platform, + backend: lease.backend, + device: backend.describe(lease), + installedBuild: lease.installedBuild, + expiresAt: null, + renewed, + }; +} + +export function leaseLine(view: LeaseView): string { + return `device ${view.device} ${view.backend} ${view.renewed ? 'renewed' : 'leased by this worktree'} installed ${view.installedBuild ?? 'nothing'}`; +} + +const BUILD_DENYLIST = [/\.md$/i, /(^|\/)\.claude\//, /(^|\/)docs\//, /(^|\/)\.verify\//]; + +export async function computeBuildKey(host: HostAdapter, platform: Platform, backend: BackendKind, worktree: string): Promise { + const listed = await run('git', ['ls-files', '-z', '--cached', '--others', '--exclude-standard', '--', ...host.buildInputs(platform, backend)], { cwd: worktree }); + if (listed.code !== 0) throw new VerifyFailure('NOT_READY', `git ls-files failed: ${listed.stderr.trim()}`, 'run {cli} from inside a git worktree'); + const files = [...new Set(listed.stdout.split('\0').filter((f) => f.length > 0 && !BUILD_DENYLIST.some((re) => re.test(f))))].sort(); + const hash = createHash('sha256'); + for (const file of files) { + const path = join(worktree, file); + hash.update(file).update('\0'); + hash.update(existsSync(path) ? readFileSync(path) : 'deleted').update('\0'); + } + return `${platform}-${hash.digest('hex').slice(0, 12)}` as BuildKey; +} + +function buildDir(workspace: Workspace, key: BuildKey): ScratchPath { + return join(workspace.buildsDir(), key) as ScratchPath; +} + +export function readBuiltApp(workspace: Workspace, key: BuildKey): LocalBuild | null { + const file = join(buildDir(workspace, key), 'build.json'); + if (!existsSync(file)) return null; + const app = JSON.parse(readFileSync(file, 'utf8')) as LocalBuild; + return existsSync(app.path) ? app : null; +} + +async function ensureBuild(host: HostAdapter, platform: Platform, workspace: Workspace, backend: DeviceBackend, progress: (line: string) => void): Promise<{ app: BuiltApp; view: BuildView }> { + const key = await computeBuildKey(host, platform, backend.kind, workspace.worktree); + const existing = readBuiltApp(workspace, key); + if (existing !== null) return { app: existing, view: { platform, key, source: existing.source, reused: true, seconds: 0 } }; + const started = Date.now(); + progress(`build ${key} local building...`); + const into = buildDir(workspace, key); + mkdirSync(into, { recursive: true }); + const app = await host.build(platform, key, into, progress); + writeFileSync(join(into, 'build.json'), `${JSON.stringify(app, null, 2)}\n`); + return { app, view: { platform, key, source: 'local', reused: false, seconds: Math.round((Date.now() - started) / 1000) } }; +} + +function closePending(workspace: Workspace, platform: Platform): void { + for (const entry of workspace.unclosedEntries()) { + if ((entry.kind === 'lease-intent' || entry.kind === 'lease-held') && entry.platform === platform) { + workspace.append({ id: newEntryId(), kind: 'done', ref: entry.id }); + } + } +} + +export async function releaseLease(workspace: Workspace, backend: DeviceBackend, lease: Lease): Promise { + await backend.release(lease); + workspace.clearLease(lease.platform); + closePending(workspace, lease.platform); +} + +export async function ensureLease( + lock: AcquireLock, + requested: BackendKind | undefined, + workspace: Workspace, + host: HostAdapter, + options: { readonly waitSeconds: number; readonly progress: (line: string) => void; readonly instances: Instances; readonly retryWith: string }, +): Promise { + const { platform } = lock; + const held = workspace.readLease(platform); + if (held !== null && requested !== undefined && held.backend !== requested) { + throw new VerifyFailure('NOT_READY', `this worktree holds a ${held.backend} ${platform} lease, not ${requested}`, '{cli} down'); + } + const choice = selectBackend(host, platform, requested, held); + const { backend } = choice; + options.progress(`backend ${describeChoice(choice)}`); + for (const stale of await backend.reapable()) { + options.progress(`reap ${backend.describe(stale)} (owner process and worktree are gone)`); + await backend.release(stale); + } + await finishOrphanLedgers(workspace.home, resolve(workspace.worktree), options.instances, options.progress); + + const { app, view: build } = await ensureBuild(host, platform, workspace, backend, options.progress); + const builtBy = build.reused ? 'reused' : `built in ${build.seconds}s`; + options.progress(`build ${build.key} ${build.source} ${builtBy}`); + + let lease: Lease | null = held; + let renewed = false; + const state = lease === null ? 'held' : await backend.check(lease); + if (lease !== null && state !== 'held') { + options.progress(`${'lost'.padEnd(7)} ${backend.describe(lease)} renewing`); + await releaseLease(workspace, backend, lease); + lease = null; + renewed = true; + } + if (lease === null) { + for (const orphan of await backend.reapable(workspace.worktree)) { + options.progress(`reap ${backend.describe(orphan)} (claimed by this worktree with no lease file)`); + await backend.release(orphan); + } + const intent = { id: newEntryId(), kind: 'lease-intent' as const, platform, backend: backend.kind, worktree: workspace.worktree }; + workspace.append(intent); + const acquired = await backend.acquire({ platform, worktree: workspace.worktree, waitSeconds: options.waitSeconds, app, retryWith: options.retryWith, progress: options.progress }); + workspace.writeLease(acquired); + workspace.append({ + id: newEntryId(), + kind: 'lease-held', + platform, + backend: backend.kind, + sessionId: null, + deviceId: acquired.deviceId, + }); + workspace.append({ id: newEntryId(), kind: 'done', ref: intent.id }); + lease = acquired; + } + + if (lease.installedBuild !== app.key) { + const target = lease; + options.progress(`install ${app.key} on ${backend.describe(target)}`); + const wait = { + seconds: options.waitSeconds, + busyFix: `let the run in this worktree finish, or rerun with a wait: ${options.retryWith}`, + onWait: (owner: ProcessRef) => + options.progress(`wait another {cli} run in this worktree (pid ${owner.pid}) is driving the device; waiting up to ${options.waitSeconds}s to install`), + }; + lease = { ...(await workspace.withDevice(platform, wait, () => backend.install(target, app, options.progress))), installedBuild: app.key }; + workspace.writeLease(lease); + } + return { lease, backend, app, build, view: leaseView(backend, lease, renewed) }; +} diff --git a/integration/expo-native/src/core/driver.ts b/integration/expo-native/src/core/driver.ts new file mode 100644 index 00000000000..dc55d61e12a --- /dev/null +++ b/integration/expo-native/src/core/driver.ts @@ -0,0 +1,438 @@ +import { createHash } from 'node:crypto'; +import { closeSync, existsSync, mkdirSync, openSync, readFileSync, readSync, readdirSync, rmSync, statSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { redact } from '../../specs/support/secret.ts'; +import { isAlive } from './exec.ts'; +import { count } from './state.ts'; +import type { EvidencePath } from './types.ts'; + +export type Registration = + | { readonly kind: 'none' } + | { readonly kind: 'unreadable' } + | { readonly kind: 'running'; readonly pid: number } + | { readonly kind: 'replaceable'; readonly pid: number } + | { readonly kind: 'stuck'; readonly pid: number }; + +type Alive = (pid: number) => boolean; + +const registrationFile = (stateDir: string): string => join(stateDir, 'daemon.json'); + +export function readRegistration(stateDir: string, alive: Alive = isAlive): Registration { + const file = registrationFile(stateDir); + if (!existsSync(file)) return { kind: 'none' }; + let raw: { readonly pid?: unknown; readonly processStartTime?: unknown } | null; + try { + raw = JSON.parse(readFileSync(file, 'utf8')) as typeof raw; + } catch { + return { kind: 'unreadable' }; + } + if (typeof raw !== 'object' || raw === null) return { kind: 'unreadable' }; + const { pid } = raw; + if (typeof pid !== 'number' || !Number.isInteger(pid) || pid <= 0) return { kind: 'unreadable' }; + if (alive(pid)) return { kind: 'running', pid }; + return typeof raw.processStartTime === 'string' && raw.processStartTime.trim() !== '' ? { kind: 'replaceable', pid } : { kind: 'stuck', pid }; +} + +export function clearStuckRegistration(stateDir: string, alive: Alive = isAlive): Registration { + const found = readRegistration(stateDir, alive); + if (found.kind === 'stuck') rmSync(registrationFile(stateDir), { force: true }); + return found; +} + +function describeRegistration(found: Registration): string { + switch (found.kind) { + case 'none': + return 'there was no daemon.json, so no agent-device daemon was registered'; + case 'unreadable': + return 'daemon.json could not be read as the registration of a daemon'; + case 'running': + return `daemon.json named pid ${found.pid}, which was running`; + case 'replaceable': + return `daemon.json named pid ${found.pid}, which was not running, and it carried a start time, so agent-device replaces it by itself`; + case 'stuck': + return `daemon.json named pid ${found.pid}, which was not running, and it carried no start time, which is what a daemon that fails to start leaves; agent-device refuses that registration and never removes it`; + default: { + const exhaustive: never = found; + return exhaustive; + } + } +} + +export const LEFT_OUT = ''; +export const linesLeftOut = (lines: number): string => `<${count(lines, 'line')} left out>`; + +const NUMBER = String.raw`-?\d+(?:\.\d+)?`; +const MOMENT = String.raw`\d{4}-\d\d-\d\d \d\d:\d\d:\d\d\.\d+`; +const ELEMENT_TYPE = + '(?:Alert|Application|Button|Cell|CheckBox|CollectionView|Image|Keyboard|Link|MenuItem|Picker|ScrollView|SearchField|SecureTextField|SegmentedControl|Sheet|Slider|StaticText|Stepper|Switch|TabBar|Table|TextField|TextView|Window)'; +const FRAME = String.raw`\{\{${NUMBER}, ${NUMBER}\}, \{${NUMBER}, ${NUMBER}\}\}`; +const ELEMENT = String.raw`(?:(?:"(.*)" )?${ELEMENT_TYPE}(?: \(Element at index \d+\)| \(First Match\)| at ${FRAME})?|Application '(.*)')`; +const QUERY = String.raw`(?:Descendants matching type ${ELEMENT_TYPE}|Element at index \d+|Elements (?:containing elements )?matching predicate (.*))`; +const RUNNER = String.raw`${MOMENT}[+-]\d{4} AgentDeviceRunnerUITests-Runner\[\d+:\d+\] `; +const XCODEBUILD = String.raw`${MOMENT} xcodebuild\[\d+:\d+\] \[MT\] IDETestOperationsObserverDebug: `; +const FAILURE = + 'Could not match active AX application for XCTest application|Error getting element frame kAXErrorInvalidUIElement|timed out while running query-sweep snapshot on the XCTest main thread|timed out while reading snapshot viewport on the XCTest main thread'; + +const whole = (shape: string): RegExp => new RegExp(`^(?:${shape})$`, 'd'); + +const ACTIVITIES: readonly RegExp[] = [ + String.raw`Find the ${ELEMENT}(?: \(retry \d+\))?`, + String.raw`Find: ${QUERY}`, + String.raw`Checking existence of \`${ELEMENT}\``, + String.raw`Check for interrupting elements affecting ${ELEMENT}`, + String.raw`Get all elements bound by index for: ${QUERY}`, + String.raw`Get number of matches for: ${QUERY}`, + String.raw`Tap ${ELEMENT}\[${NUMBER}, ${NUMBER}\](?: -> \(${NUMBER}, ${NUMBER}\))?`, + String.raw`Type (.*)`, + String.raw`Synthesize event|Set Up|Tear Down|Collecting debug information to assist test failure triage|Interface orientation changed to Portrait`, + String.raw`Start Test at ${MOMENT}`, + String.raw`Requesting snapshot of accessibility hierarchy for app with pid \d+`, + String.raw`Wait for (?:com\.apple\.springboard|(.*)) to idle`, + String.raw`Ignoring failure to (?:get hierarchy for remote element in process \d+ \(Error getting main window kAXErrorInvalidUIElement\)|(.*))`, +].map(whole); + +const LINES: readonly RegExp[] = [ + '', + String.raw`${RUNNER}\[DEBUG-\d+\] synthesize posted \d+ chars status=-?\d+ tookMs=\d+`, + String.raw`${RUNNER}\[Default\] Running tests\.\.\.`, + String.raw`${RUNNER}\[connection\] (?:Connection interrupted: will attempt to reconnect|Connection invalidated!|XPC message send failed|Handshake aborted as the connection has been invalidated|Handshake failed with error: (.*))`, + String.raw`${XCODEBUILD}(?:${NUMBER} elapsed -- Testing started completed\.|${NUMBER} sec, \+${NUMBER} sec -- (?:start|end))`, + String.raw`Test Suite '(.*)' (?:started|passed|failed) at ${MOMENT}\.`, + String.raw`Test Case '(.*)' (?:started|(?:passed|failed) \(${NUMBER} seconds\))\.`, + String.raw`\t Executed \d+ tests?, with \d+ failures? \(\d+ unexpected\) in ${NUMBER} \(${NUMBER}\) seconds`, + String.raw`Testing started|Testing failed:|Failing tests:|\tRunnerTests\.testCommand\(\)|\*\* (?:TEST EXECUTE FAILED|TEST EXECUTE SUCCEEDED|BUILD INTERRUPTED) \*\*`, + String.raw`(.*): error: -\[AgentDeviceRunnerUITests\.RunnerTests testCommand\] : (?:Failed to get matching snapshot: |Failed to resolve query: )?(.*)`, + String.raw`\tNo matching device \((.*)\) in set at (.*)`, + String.raw`AGENT_DEVICE_DAEMON_(?:HTTP_)?PORT=\d+`, + String.raw`Daemon registration (.*); exiting\.`, + String.raw`Daemon error: (.*)`, +].map(whole); + +const ENDS_A_LINE_INSIDE_A_LINE = /[\r\u2028\u2029]/; +const ACTIVITY_LINE = /^([ \t]*t =[ \t]*)(\S+)([ \t]+)(.*)$/; +const READABLE_TIME = /^(?:-?\d+(?:\.\d+)?|nan)s$/; + +const RUNNER_TAGS = + 'ABANDONED_WORK_DRAINED|ACTIVATE|ACTIVATE_FACT|ACTIVATE_SKIPPED|ALERT_ACTIVATION|APP_SCREEN_UNRESOLVED|AX_SNAPSHOT_ISSUE_SUPPRESSED|BUSY|COMMAND_ACCEPTED|COMMAND_COALESCED|COMMAND_COMPLETED|COMMAND_FAILED|COORDINATE_TAP_TEXT_INPUT_PROBE_SKIPPED|DESIRED_PORT|DISPATCH_RECOVERY_SKIPPED_XCTEST_OCCUPIED|ELEMENT_TAP_IGNORED_EXCEPTION|FAST_APP_GUARD|HEADLESS_STARTUP|HOST_ACTIVATE|IDLE_KEEPALIVE|IN_APP_BACK_SKIPPED_XCTEST_ENUMERATION|IN_APP_BACK_VISUAL_VERIFICATION|KEYBOARD_AVOIDING_DRAG|KEYBOARD_BAND_FACT|KEYBOARD_RETURN_IGNORED_EXCEPTION|KEYBOARD_RETURN_TARGET_IGNORED_EXCEPTION|KEYBOARD_STABILITY|LISTENER_FAILED|LISTENER_READY|MAIN_THREAD_WORK_ABANDONED|MAIN_THREAD_WORK_DRAINED|PORT|PORT_NOT_SET|POST_SNAPSHOT_DELAY_MARK_FAILED|POST_SNAPSHOT_DELAY_MARK_SKIPPED_XCTEST_OCCUPIED|PRIVATE_AX_CUSTOM_ACTIONS|PRIVATE_AX_CUSTOM_ACTIONS_READ_TIMEOUT|PRIVATE_AX_DEEP_EXTENSION|PRIVATE_AX_DEEP_EXTENSION_MISS|PRIVATE_AX_DEPTH_MEMORY_CLEARED|PRIVATE_AX_DEPTH_REMEMBERED|PRIVATE_AX_SNAPSHOT_BUDGET_EXHAUSTED|PRIVATE_AX_SNAPSHOT_DEPTH_RETRY|PRIVATE_AX_SNAPSHOT_FAILED|PRIVATE_AX_SNAPSHOT_SPARSE|PRIVATE_AX_SNAPSHOT_USED|PRIVATE_AX_VIEWPORT_FALLBACK|READ_TARGET_NOT_RUNNING|RECORD_START|RECORD_STOP_FAILED|REPAIR_TEXT_ENTRY|RETRY|SCREEN_CAPTURE|SCROLL_VIEWPORT|SEND_FAILED|SNAPSHOT_AX_UNAVAILABLE|SNAPSHOT_BACKEND_FAILED|SNAPSHOT_FLAT_FALLBACK_DEADLINE|SNAPSHOT_FLAT_IGNORED_EXCEPTION|SNAPSHOT_PLAN_BUDGET_EXHAUSTED|SNAPSHOT_PROJECTION_MISMATCH|SNAPSHOT_QUERY_IGNORED_EXCEPTION|SNAPSHOT_RECOVERED|SNAPSHOT_STATE_DEFERRED_XCTEST_OCCUPIED|SNAPSHOT_STATE_FAILED|SNAPSHOT_TAB_FALLBACK_IGNORED_EXCEPTION|SNAPSHOT_TIER_DEADLINE_EXHAUSTED|SNAPSHOT_TIER_SKIPPED_XCTEST_OCCUPIED|SNAPSHOT_XCTEST_CHANNEL_DEFERRED|SNAPSHOT_XCTEST_CHANNEL_PENALIZED|SNAPSHOT_XCTEST_CHANNEL_PENALTY_CLEARED|SNAPSHOT_XCTEST_CHANNEL_PROBE_BOUNDED|SYNTHESIZED_DISPATCH|SYNTHESIZED_GESTURE_POLICY|SYNTHESIZED_RECORD|SYSTEM_MODAL_PROBE_ABORTED|SYSTEM_MODAL_PROBE_DEADLINE|SYSTEM_MODAL_PROBE_SKIPPED|TARGET_CACHE_INVALIDATE|TARGET_CACHE_REFRESH|TEXT_ENTRY_CLEAR|TEXT_ENTRY_INPUT_REMOVED_AFTER_DELIVERY|TEXT_ENTRY_PHASE|TEXT_ENTRY_REPAIR_REFUSED|TEXT_ENTRY_ROUTE|TEXT_ENTRY_UNCONFIRMED|TEXT_INPUT_PROBE_UNAVAILABLE|WAITING|WAIT_RESULT|WEDGED'; +const RUNNER_EVENT = new RegExp(String.raw`^(${RUNNER}AGENT_DEVICE_RUNNER_(?:${RUNNER_TAGS})(?![A-Z0-9_]))(.*)$`); +const VALUE_ALONE = new RegExp(String.raw`^(?:${NUMBER}|${FAILURE}|XCTWaiterResult\(rawValue: \d+\))$`); +const HIDDEN_FIELDS: ReadonlySet = new Set(['bundle']); +const numbers = (...keys: readonly string[]): Record => Object.fromEntries(keys.map((key) => [key, NUMBER])); +const tuples = (...keys: readonly string[]): Record => Object.fromEntries(keys.map((key) => [key, String.raw`\(${NUMBER}(?:,${NUMBER})*\)`])); +const FIELDS: ReadonlyMap = new Map( + Object.entries({ + ...numbers('ok', 'chars', 'commandChars', 'elapsedMs', 'durationMs', 'orientation', 'displayID', 'interfaceOrientation', 'state', 'depth', 'nodes', 'extended', 'slice', 'abandonedForSeconds', 'expectedLength', 'observedLength', 'repaired', 'swipeHeight'), + ...tuples('point', 'reference', 'start', 'end', 'frame'), + bundle: String.raw`\S+`, + commandId: String.raw`runner-[0-9A-Fa-f]{8}-(?:[0-9A-Fa-f]{4}-){3}[0-9A-Fa-f]{12}`, + command: 'snapshot|tap|targetReset|type|shutdown|alert|gestureViewport|gesture|scroll|keyboardDismiss', + route: 'synthesized-first-responder|xctest-application-fallback', + kind: 'absent|visible|unmeasurable|tap|drag|coordinateTap|scroll|synthesizedDrag', + phase: 'focus|initial-resolve|total|verify|type-delayed|type-first|warmup|type-remaining|clear|type-all', + mode: 'append|replacement', + name: 'agent-device-tap|agent-device-swipe|agent-device-controlled-scroll', + axHealth: 'healthy|unknown', + frameSource: 'window', + keyboardPolicy: 'never|requiredWhenAvailable', + fallbackPolicy: 'xctestCoordinateAllowed|privateSynthesisRequired|xctestCoordinateWhenAccessibilityAvailable', + fallbackAllowed: 'true|false', + fallbackAttempted: 'true|false', + backend: 'private-ax|queries|tree', + tier: 'queries', + operation: 'query_sweep|keyboard_band|command_execution|snapshot_viewport|accessibility_health', + action: 'dismiss', + decision: 'noKeyboard|unobstructed|avoided', + keyboardMinY: `none|${NUMBER}`, + reason: + '-|keyboard-frame-query-timeout|keyboard-frame-unusable|delivery-budget|bundle_changed|already_foreground|external_app_relaunch|xctest_recorded_failure|queries_backend_timeout|tree_backend_timeout|XCTest-backed snapshot tiers were deferred after recent slow accessibility work on this screen|snapshot returned no semantic controls or content|XCTest-backed snapshot tiers are running with a short recovery probe after recent slow accessibility work on this screen|timed out while reading snapshot viewport on the XCTest main thread|the queries backend spent its capture slice with the collection unfinished', + error: FAILURE, + }).map(([key, values]) => [key, new RegExp(`(?:${values})(?= |$)`, 'y')]), +); +const FIELD_KEY = / ([A-Za-z]+)=/y; + +const DAEMON_EVENT_START = /^\{"ts":"\d{4}-\d\d-\d\dT/; +const DAEMON_EVENT_TIME = /^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d(?:\.\d+)?Z$/; +const DAEMON_EVENT_LEVEL = /^(?:debug|info|warn|error)$/; +const DAEMON_EVENT_PHASE = /^(?:ios_runner_session_detached|ios_runner_session_detach_skipped)$/; +const DAEMON_EVENT_DATA: ReadonlyMap = new Map( + Object.entries({ lane: /^simulator$/, reason: /^runner_never_served_a_command$/, runnerPid: /^\d+$/, port: /^\d+$/, outstandingCharges: /^\d+$/, hasAbandonedCharges: /^(?:true|false)$/ }), +); + +interface Shown { + readonly text: string; + readonly partly: boolean; +} + +type Unknown = 'activity' | 'line'; + +function withFreeTextLeftOut(match: RegExpExecArray): Shown { + let text = ''; + let from = 0; + let partly = false; + if (match.indices === undefined) throw new Error('a shape of the copy does not say where its free text stands'); + for (const span of match.indices.slice(1)) { + if (span === undefined) continue; + text += `${match[0].slice(from, span[0])}${LEFT_OUT}`; + from = span[1]; + partly = true; + } + return { text: text + match[0].slice(from), partly }; +} + +function knownShape(shapes: readonly RegExp[], text: string): Shown | null { + for (const shape of shapes) { + const match = shape.exec(text); + if (match !== null) return withFreeTextLeftOut(match); + } + return null; +} + +function runnerFieldsShown(fields: string): Shown { + if (fields === '') return { text: '', partly: false }; + if (fields.startsWith('=')) return VALUE_ALONE.test(fields.slice(1)) ? { text: fields, partly: false } : { text: `=${LEFT_OUT}`, partly: true }; + let text = ''; + let partly = false; + for (let at = 0; at < fields.length; ) { + FIELD_KEY.lastIndex = at; + const key = FIELD_KEY.exec(fields)?.[1]; + const values = key === undefined ? undefined : FIELDS.get(key); + if (key === undefined || values === undefined) return { text: `${text} ${LEFT_OUT}`, partly: true }; + values.lastIndex = FIELD_KEY.lastIndex; + const value = values.exec(fields)?.[0]; + if (value === undefined) return { text: `${text} ${key}=${LEFT_OUT}`, partly: true }; + const hidden = HIDDEN_FIELDS.has(key); + text += ` ${key}=${hidden ? LEFT_OUT : value}`; + partly ||= hidden; + at = values.lastIndex; + } + return { text, partly }; +} + +function daemonEventShown(line: string): Shown | null { + if (!DAEMON_EVENT_START.test(line)) return null; + let event: unknown; + try { + event = JSON.parse(line); + } catch { + return null; + } + if (typeof event !== 'object' || event === null || Array.isArray(event)) return null; + const { ts, level, phase, data } = event as Readonly>; + if (typeof ts !== 'string' || !DAEMON_EVENT_TIME.test(ts) || typeof level !== 'string' || !DAEMON_EVENT_LEVEL.test(level) || typeof phase !== 'string' || !DAEMON_EVENT_PHASE.test(phase)) return null; + const all = Object.entries(typeof data === 'object' && data !== null ? data : {}); + const known = all.filter(([key, value]) => ['number', 'boolean', 'string'].includes(typeof value) && DAEMON_EVENT_DATA.get(key)?.test(String(value)) === true); + const fieldsLeftOut = Object.keys(event).filter((key) => !['ts', 'level', 'phase', 'data'].includes(key)).length + all.length - known.length; + return { text: `${ts} ${level} ${phase}${known.map(([key, value]) => ` ${key}=${String(value)}`).join('')} fieldsLeftOut=${fieldsLeftOut}`, partly: fieldsLeftOut > 0 }; +} + +function lineShown(line: string): Shown | Unknown { + if (ENDS_A_LINE_INSIDE_A_LINE.test(line)) return 'line'; + const activity = ACTIVITY_LINE.exec(line); + if (activity !== null) { + const known = knownShape(ACTIVITIES, activity[4] ?? ''); + if (known === null) return 'activity'; + const time = READABLE_TIME.test(activity[2] ?? '') ? activity[2] : LEFT_OUT; + return { text: `${activity[1]}${time}${activity[3]}${known.text}`, partly: known.partly || time === LEFT_OUT }; + } + const event = RUNNER_EVENT.exec(line); + if (event !== null) { + const fields = runnerFieldsShown(event[2] ?? ''); + return { text: `${event[1]}${fields.text}`, partly: fields.partly }; + } + return knownShape(LINES, line) ?? daemonEventShown(line) ?? 'line'; +} + +export interface Scrubbed { + readonly text: string; + readonly lines: number; + readonly whole: number; + readonly partly: number; + readonly unknownActivities: number; + readonly unknownLines: number; +} + +export function scrubDriverLog(log: string): Scrubbed { + const endsALine = log.endsWith('\n'); + const lines = log === '' ? [] : (endsALine ? log.slice(0, -1) : log).split('\n'); + const tally = { whole: 0, partly: 0, activity: 0, line: 0 }; + const kept: string[] = []; + let leftOut = 0; + const sayWhatWasLeftOut = (): void => { + if (leftOut > 0) kept.push(linesLeftOut(leftOut)); + leftOut = 0; + }; + for (const line of lines) { + const shown = lineShown(line); + if (typeof shown === 'string') { + tally[shown] += 1; + leftOut += 1; + continue; + } + sayWhatWasLeftOut(); + tally[shown.partly ? 'partly' : 'whole'] += 1; + kept.push(shown.text); + } + sayWhatWasLeftOut(); + return { text: redact(kept.length === 0 ? '' : `${kept.join('\n')}${endsALine ? '\n' : ''}`), lines: lines.length, whole: tally.whole, partly: tally.partly, unknownActivities: tally.activity, unknownLines: tally.line }; +} + +export const DRIVER_LOG_LIMIT_BYTES = 4 * 1024 * 1024; +const HEAD_BYTES = 4096; +const NEWLINE = 0x0a; + +interface Mark { + readonly size: number; + readonly head: string; + readonly endsALine: boolean; +} + +type Since = Mark | 'the file did not exist' | 'the start of the file' | 'the file could not be read'; + +const isMissing = (error: unknown): boolean => (error as NodeJS.ErrnoException).code === 'ENOENT'; +const nameOf = (error: unknown): string => (error instanceof Error ? ((error as NodeJS.ErrnoException).code ?? error.name) : 'an error'); + +function readRange(file: string, from: number, to: number): Buffer { + const bytes = Buffer.alloc(Math.max(0, to - from)); + const fd = openSync(file, 'r'); + try { + let read = 0; + for (let got = -1; read < bytes.length && got !== 0; read += got) got = readSync(fd, bytes, read, bytes.length - read, from + read); + return bytes.subarray(0, read); + } finally { + closeSync(fd); + } +} + +const headOf = (file: string, size: number): string => createHash('sha256').update(readRange(file, 0, Math.min(size, HEAD_BYTES))).digest('hex'); + +function markOf(file: string): Mark | 'the file did not exist' | 'the file could not be read' { + try { + const { size } = statSync(file); + return { size, head: headOf(file, size), endsALine: size === 0 || readRange(file, size - 1, size)[0] === NEWLINE }; + } catch (error) { + return isMissing(error) ? 'the file did not exist' : 'the file could not be read'; + } +} + +interface Part { + readonly copy: Scrubbed; + readonly bytes: number; + readonly leftOut: number; + readonly is: 'what the run added' | 'written during the run' | 'as it was when its registration was removed'; +} + +type Scrub = (log: string) => Scrubbed; + +function partOf(file: string, since: Exclude, scrub: Scrub): Part | null { + if (!existsSync(file)) return null; + const { size } = statSync(file); + const grewInPlace = typeof since === 'object' && since.size > 0 && size >= since.size && headOf(file, since.size) === since.head; + const start = grewInPlace ? since.size : 0; + const from = Math.max(start, size - DRIVER_LOG_LIMIT_BYTES); + const read = readRange(file, from, size); + const startsInALine = from > start || (grewInPlace && !since.endsALine); + const kept = startsInALine ? (read.includes(NEWLINE) ? read.subarray(read.indexOf(NEWLINE) + 1) : read.subarray(read.length)) : read; + return { + copy: scrub(kept.toString('utf8')), + bytes: size - start, + leftOut: size - start - kept.length, + is: since === 'the start of the file' ? 'as it was when its registration was removed' : grewInPlace ? 'what the run added' : 'written during the run', + }; +} + +const COPIED: Readonly string>> = { + 'what the run added': (source) => `the part of ${source} that was written during this run`, + 'written during the run': (source) => `the whole of ${source}, which was written or rewritten during this run`, + 'as it was when its registration was removed': (source) => `the whole of ${source} as it was when the CLI removed that registration`, +}; + +const linesOf = (copy: Scrubbed): string => + `Of the ${count(copy.lines, 'line')} read, the copy keeps ${copy.whole} as written and ${copy.partly} with a label, typed text, name or message left out, and it leaves out ${copy.unknownActivities + copy.unknownLines} whose shape it does not know: ${copy.unknownActivities} with an XCTest activity and ${copy.unknownLines} without.`; + +function sessionsOf(stateDir: string): readonly string[] | 'the sessions could not be read' { + try { + return readdirSync(join(stateDir, 'sessions')).sort(); + } catch (error) { + return isMissing(error) ? [] : 'the sessions could not be read'; + } +} + +const runnerLogOf = (stateDir: string, session: string): string => join(stateDir, 'sessions', session, 'runner.log'); +const SESSION_NAME = /^[A-Za-z0-9._-]{1,64}$/; + +export interface DriverWatch { + startClean(progress: (line: string) => void): void; + daemonThatDidNotStart(): string | null; + collect(): void; +} + +export function watchDriver(stateDir: string, runDir: EvidencePath, alive: Alive = isAlive, scrub: Scrub = scrubDriverLog): DriverWatch { + const dir = join(runDir, 'driver'); + const atStart = readRegistration(stateDir, alive); + const daemonMark = markOf(join(stateDir, 'daemon.log')); + const sessionsAtStart = sessionsOf(stateDir); + const runnerMarks = typeof sessionsAtStart === 'string' ? null : new Map(sessionsAtStart.map((session) => [session, markOf(runnerLogOf(stateDir, session))] as const)); + const said: string[] = [`At the start of the run ${describeRegistration(atStart)}.`]; + let cleared: Extract | null = null; + + const keep = (name: string, source: string, since: Since, missing: string): void => { + if (since === 'the file could not be read') { + return void said.push(`${source} could not be read when the run began, so the CLI cannot tell which part of it this run wrote, and there is no ${name}.`); + } + let part: Part | null; + try { + part = partOf(join(stateDir, source), since, scrub); + } catch (error) { + return void said.push(`${source} could not be copied (${nameOf(error)}), so there is no ${name}.`); + } + if (part === null) return void said.push(missing); + if (part.bytes === 0) return void said.push(`${source} ${part.is === 'what the run added' ? 'did not grow during the run' : 'is empty'}, so there is no ${name}.`); + mkdirSync(dir, { recursive: true }); + writeFileSync(join(dir, name), part.copy.text); + const cut = part.leftOut === 0 ? '' : `, of which the first ${part.leftOut} were left out, because a copy starts at the start of a line and holds ${DRIVER_LOG_LIMIT_BYTES} bytes at most`; + said.push(`${name} is ${COPIED[part.is](source)}: ${part.bytes} bytes${cut}. ${linesOf(part.copy)}`); + }; + + return { + startClean(progress) { + if (cleared !== null) return; + const found = clearStuckRegistration(stateDir, alive); + if (found.kind !== 'stuck') return; + cleared = found; + keep('daemon-that-did-not-start.log', 'daemon.log', 'the start of the file', 'The daemon that failed to start left no daemon.log.'); + said.push(`Before a group of tests started, the CLI removed the daemon.json that named pid ${found.pid}, which was not running, so that e2e could start a new daemon. The CLI does this once in a run.`); + progress(`driver removed the agent-device registration of pid ${found.pid}, which is not running and has no start time; a daemon failed to start, and agent-device would refuse every command until the file was gone`); + }, + daemonThatDidNotStart() { + const now = readRegistration(stateDir, alive); + if (now.kind !== 'stuck') return null; + const before = cleared === null ? '' : `, as the daemon before it had (pid ${cleared.pid}), whose registration the CLI removed once in this run and does not remove a second time`; + return `agent-device's daemon did not start: it left a registration for pid ${now.pid}, which is not running${before}`; + }, + collect() { + said.push(`At the end of the run ${describeRegistration(readRegistration(stateDir, alive))}.`); + keep('daemon.log', 'daemon.log', daemonMark, 'There is no daemon.log, so no agent-device daemon on this machine wrote one.'); + const sessionsAtEnd = sessionsOf(stateDir); + if (runnerMarks === null || typeof sessionsAtEnd === 'string') { + said.push(`The sessions of agent-device could not be read ${runnerMarks === null ? 'when the run began' : 'when the run ended'}, so the CLI cannot tell which part of a runner.log this run wrote, and no runner.log is copied.`); + } else { + const withALog = sessionsAtEnd.filter((session) => existsSync(runnerLogOf(stateDir, session))); + const sessions = withALog.filter((session) => SESSION_NAME.test(session)); + if (withALog.length === 0) said.push('No session has a runner.log. The XCTest runner writes one, on iOS only.'); + if (sessions.length < withALog.length) { + said.push(`The runner.log of ${count(withALog.length - sessions.length, 'session')} is not copied, because the name of the session is not one the CLI gives: letters, digits, dots, dashes and underscores, 64 at most.`); + } + for (const session of sessions) { + const source = `sessions/${session}/runner.log`; + keep(`runner-${redact(session).replace(/[^A-Za-z0-9._-]/g, '-')}.log`, source, runnerMarks.get(session) ?? 'the file did not exist', `${source} is gone.`); + } + } + said.push( + 'These are logs of the agent-device daemon on this machine. A remote session drives its device with a daemon on the runner, and that daemon writes its logs there.', + `A copy is not the whole log. It holds a line only when the CLI knows the whole shape of the line, and it writes that line again from the fixed words of the shape, its numbers and its times. Where the log shows a label, typed text, a predicate, a name or a message, the copy shows ${LEFT_OUT}. Lines of any other shape are counted above and shown as one line such as ${linesLeftOut(2)}. Each copy then passed through the redaction the CLI applies to app.log.`, + ); + mkdirSync(dir, { recursive: true }); + writeFileSync(join(dir, 'summary.txt'), redact(`${said.join('\n')}\n`)); + }, + }; +} diff --git a/integration/expo-native/src/core/e2e.ts b/integration/expo-native/src/core/e2e.ts new file mode 100644 index 00000000000..e2364e3f8d5 --- /dev/null +++ b/integration/expo-native/src/core/e2e.ts @@ -0,0 +1,275 @@ +import { spawn } from 'node:child_process'; +import { appendFileSync, copyFileSync, existsSync, mkdirSync, readdirSync, statSync } from 'node:fs'; +import { isAbsolute, join, relative, resolve, sep } from 'node:path'; +import { INPUT_VARIABLES, inputsEnv, type Inputs } from '../../specs/support/inputs.ts'; +import { agentEnvironment } from './agent.ts'; +import type { Agent } from '../../specs/support/agent.ts'; +import { withoutClerkKeys } from './keys.ts'; +import { redact } from '../../specs/support/secret.ts'; +import { WORKSPACE_DIR, agentDeviceStateDir } from './workspace.ts'; +import { + VerifyFailure, + type E2EInvocation, + type EvidencePath, + type FeatureName, + type RunCommand, + type SpecRef, + type SpecResult, + type SpecSelection, + type SpecStatus, +} from './types.ts'; + +const SPEC_SUFFIX = '.e2e.ts'; + +function walk(dir: string): string[] { + if (!existsSync(dir)) return []; + return readdirSync(dir, { withFileTypes: true }).flatMap((entry) => { + const path = join(dir, entry.name); + return entry.isDirectory() ? walk(path) : entry.name.endsWith(SPEC_SUFFIX) ? [path] : []; + }); +} + +const toPosix = (path: string) => path.split(sep).join('/'); + +function specRef(packageDir: string, absolute: string): SpecRef | null { + const path = toPosix(relative(packageDir, absolute)); + const golden = /^specs\/golden\/([^/]+)\/.+\.e2e\.ts$/.exec(path); + if (golden !== null) return { kind: 'golden', path, feature: golden[1] as FeatureName }; + if (/^specs\/explored\/.+\.e2e\.ts$/.test(path)) return { kind: 'explored', path, feature: null }; + return null; +} + +export function resolveSpecs(packageDir: string, selection: SpecSelection, cwd: string = process.cwd()): readonly SpecRef[] { + const goldenDir = join(packageDir, 'specs', 'golden'); + const features = existsSync(goldenDir) ? readdirSync(goldenDir).filter((name) => statSync(join(goldenDir, name)).isDirectory()).sort() : []; + const toRefs = (paths: readonly string[]) => paths.flatMap((p) => specRef(packageDir, p) ?? []).sort((a, b) => a.path.localeCompare(b.path)); + + if ('all' in selection) { + const refs = toRefs(walk(goldenDir)); + if (refs.length === 0) throw new VerifyFailure('NO_SPECS', 'there are no golden specs under specs/golden/', 'write one under specs/golden//, or run an explored spec by path'); + return refs; + } + + const found = new Map(); + for (const selector of selection.selectors) { + let refs: readonly SpecRef[] = []; + if (selector.endsWith(SPEC_SUFFIX)) { + const candidates = isAbsolute(selector) ? [selector] : [resolve(cwd, selector), resolve(packageDir, selector)]; + const file = candidates.find((c) => existsSync(c)); + if (file !== undefined) { + const ref = specRef(packageDir, file); + if (ref === null) throw new VerifyFailure('NO_SPECS', `${selector} is not under specs/golden// or specs/explored/`, 'move the spec under specs/explored/ and run it by that path'); + refs = [ref]; + } + } else if (selector.includes('/')) { + const [feature, spec] = selector.split('/', 2); + const file = join(goldenDir, feature ?? '', `${(spec ?? '').replace(/\.e2e\.ts$/, '')}${SPEC_SUFFIX}`); + if (existsSync(file)) refs = toRefs([file]); + } else if (features.includes(selector)) { + refs = toRefs(walk(join(goldenDir, selector))); + } + if (refs.length === 0) { + throw new VerifyFailure( + 'NO_SPECS', + `no specs match ${selector}`, + features.length > 0 ? `name a feature (${features.join(', ')}), one of its specs as /, or a path to a .e2e.ts file under specs/explored/` : 'pass a path to a .e2e.ts file under specs/explored/', + ); + } + for (const ref of refs) found.set(ref.path, ref); + } + return [...found.values()]; +} + +export function e2eOutputDir(runDir: EvidencePath, index: number): EvidencePath { + return join(runDir, index === 0 ? 'e2e' : `e2e-${index + 1}`) as EvidencePath; +} + +export function planE2E(inputs: Inputs, specs: readonly SpecRef[], command: RunCommand, packageDir: string, outputDir: EvidencePath): E2EInvocation { + const output = toPosix(relative(packageDir, outputDir)); + const args = [ + 'run', + ...specs.map((s) => s.path), + '--config', + 'e2e.config.ts', + '--target', + inputs.target.platform, + '--output', + output, + '--reporter', + 'list,markdown,junit', + '--retries', + String(command.retries), + ...(command.grep === undefined ? [] : ['--grep', command.grep]), + '--pass-with-no-tests', + ]; + return { args, env: { ...inputsEnv(inputs), AGENT_DEVICE_STATE_DIR: agentDeviceStateDir(join(packageDir, WORKSPACE_DIR)), E2E_TELEMETRY_DISABLED: '1' } }; +} + +const withoutInputs = (env: NodeJS.ProcessEnv): NodeJS.ProcessEnv => Object.fromEntries(Object.entries(env).filter(([name]) => !INPUT_VARIABLES.includes(name))); + +export async function invokeE2E(invocation: E2EInvocation, log: EvidencePath, packageDir: string, onLine: (line: string) => void, agent: Agent | null): Promise<{ readonly exitCode: number }> { + const bin = join(packageDir, 'node_modules', '.bin', 'e2e'); + if (!existsSync(bin)) throw new VerifyFailure('NOT_READY', 'the pinned e2e is not installed', `cd ${packageDir} && npm ci`); + return new Promise((resolvePromise) => { + const child = spawn(bin, [...invocation.args], { + cwd: packageDir, + env: { ...withoutInputs(withoutClerkKeys(process.env)), ...invocation.env, ...agentEnvironment(agent), NO_COLOR: '1' }, + stdio: ['ignore', 'pipe', 'pipe'], + }); + const pipe = (stream: NodeJS.ReadableStream) => { + let buffered = ''; + const flush = (line: string) => { + const safe = redact(line); + appendFileSync(log, `${safe}\n`); + onLine(safe); + }; + stream.on('data', (chunk: Buffer) => { + buffered += chunk.toString(); + const lines = buffered.split('\n'); + buffered = lines.pop() ?? ''; + lines.forEach(flush); + }); + stream.on('end', () => { + if (buffered.length > 0) flush(buffered); + }); + }; + pipe(child.stdout); + pipe(child.stderr); + child.on('error', () => resolvePromise({ exitCode: 127 })); + child.on('close', (code) => resolvePromise({ exitCode: code ?? 1 })); + }); +} + +interface WireError { + readonly message?: string; +} +interface WireArtifact { + readonly id?: string; + readonly kind?: string; + readonly path?: string; + readonly producer?: { readonly kind?: string; readonly stepId?: string }; +} +interface WireStep { + readonly id?: string; + readonly api?: string; + readonly label?: string; +} +interface WireAttempt { + readonly status?: string; + readonly durationMs?: number; + readonly error?: WireError; + readonly failure?: { readonly screen?: string; readonly screenshot?: string }; + readonly artifacts?: readonly WireArtifact[]; + readonly steps?: readonly WireStep[]; +} +interface WireResult { + readonly id?: string; + readonly kind?: string; + readonly titlePath?: readonly string[]; + readonly file?: string; + readonly platform?: string; + readonly tags?: readonly string[]; + readonly status?: string; + readonly skip?: { readonly cause?: string; readonly reason?: string }; + readonly attempts?: readonly WireAttempt[]; +} + +function wireResults(reportJson: unknown): readonly WireResult[] { + const report = reportJson as { schemaVersion?: unknown; run?: { results?: unknown } } | null; + if (report?.schemaVersion !== 'report-1' || !Array.isArray(report.run?.results)) { + throw new VerifyFailure('E2E_CRASHED', 'e2e wrote a report this CLI cannot read (expected schemaVersion report-1)', 'check e2e-pins with `{cli} doctor`'); + } + return report.run.results as WireResult[]; +} + +const STATUS: Readonly> = { + passed: 'passed', + failed: 'failed', + 'timed-out': 'failed', + flaky: 'flaky', + interrupted: 'interrupted', + skipped: 'skipped', +}; + +function platformSkip(reason: string | undefined): string { + const declared = /platforms \[([^\]]*)\]/.exec(reason ?? '')?.[1]; + return declared === undefined ? `skipped: ${reason ?? 'other platform'}` : `skipped: ${declared.split(/,\s*/).join(' and ')} only`; +} + +export function parseE2EReport(reportJson: unknown, specs: readonly SpecRef[], outputDir: EvidencePath): readonly SpecResult[] { + const failuresDir = join(outputDir, 'failures'); + const pages = existsSync(failuresDir) ? readdirSync(failuresDir) : []; + const selected = new Set(specs.map((s) => s.path)); + return wireResults(reportJson) + .filter((r) => (r.kind === 'test' || r.kind === 'setup') && selected.has(r.file ?? '')) + .map((r): SpecResult => { + const file = r.file ?? ''; + const spec = specs.find((s) => s.path === file)!; + const attempts = r.attempts ?? []; + const shown = attempts.findLast((attempt) => attempt.error !== undefined) ?? attempts.at(-1); + const notRun = r.status === 'skipped' && r.skip?.cause !== 'filtered' && r.skip?.cause !== 'platform-unavailable'; + const status = notRun ? 'failed' : (STATUS[r.status ?? ''] ?? 'failed'); + const page = r.id === undefined ? undefined : pages.find((p) => p.endsWith(`-${r.id!.slice(0, 8)}.md`)); + const artifactPath = (id: string | undefined): EvidencePath | null => { + const path = id === undefined ? undefined : shown?.artifacts?.find((a) => a.id === id)?.path; + return path === undefined ? null : (join(outputDir, 'artifacts', path) as EvidencePath); + }; + const screenPath = artifactPath(shown?.failure?.screen); + let skipReason: string | null = null; + let skippedBy: SpecResult['skippedBy'] = null; + if (status === 'skipped') { + if (r.skip?.cause === 'platform-unavailable') skippedBy = 'platform'; + skipReason = r.skip?.cause === 'platform-unavailable' ? platformSkip(r.skip.reason) : `${r.skip?.cause ?? 'skipped'}: ${r.skip?.reason ?? ''}`.trim(); + } + const message = notRun ? `not run: ${r.skip?.cause ?? 'skipped'} ${r.skip?.reason ?? ''}`.trim() : shown?.error?.message; + return { + spec, + title: (r.titlePath ?? []).join(' > '), + platform: r.platform === 'android' ? 'android' : 'ios', + status, + seconds: Math.round(attempts.reduce((sum, a) => sum + (a.durationMs ?? 0), 0) / 100) / 10, + attempts: attempts.length, + error: message === undefined ? null : redact(message.split('\n').filter((line) => line.trim().length > 0).join('; ')), + skipReason, + skippedBy, + tags: r.tags ?? [], + failurePage: page === undefined ? null : (join(failuresDir, page) as EvidencePath), + failureScreen: screenPath !== null && existsSync(screenPath) ? (screenPath as EvidencePath) : null, + failureScreenshot: artifactPath(shown?.failure?.screenshot), + }; + }); +} + +export function collectScreenshots(reportJson: unknown, runDir: EvidencePath, outputDir: EvidencePath): readonly { readonly label: string; readonly path: EvidencePath }[] { + const out = new Map(); + const dir = join(runDir, 'screenshots'); + for (const result of wireResults(reportJson)) { + for (const attempt of result.attempts ?? []) { + const steps = new Map((attempt.steps ?? []).map((s) => [s.id, s])); + for (const artifact of attempt.artifacts ?? []) { + if (artifact.kind !== 'screenshot' || artifact.path === undefined) continue; + const step = artifact.producer?.stepId === undefined ? undefined : steps.get(artifact.producer.stepId); + if (step?.api !== 'app.screenshot' || !step.label) continue; + const source = join(outputDir, 'artifacts', artifact.path); + if (!existsSync(source)) continue; + const label = step.label.replace(/[^A-Za-z0-9._-]/g, '-'); + mkdirSync(dir, { recursive: true }); + const target = join(dir, `${label}.png`) as EvidencePath; + copyFileSync(source, target); + out.set(label, target); + } + } + } + return [...out].map(([label, path]) => ({ label, path })); +} + +export function assertSomethingRan(results: readonly SpecResult[], selection: string): 'ran' | 'all-left-out' { + if (results.some((r) => r.status !== 'skipped')) return 'ran'; + if (results.some((r) => r.skippedBy !== null)) return 'all-left-out'; + const reasons = [...new Set(results.map((r) => r.skipReason).filter((x): x is string => x !== null))]; + throw new VerifyFailure( + 'NO_SPECS', + `no test ran for ${selection}${reasons.length === 0 ? ': the selection registered no tests' : `: ${reasons.join('; ')}`}`, + 'check the --grep pattern and the spec files; {cli} run runs every test in it', + ); +} diff --git a/integration/expo-native/src/core/evidence.ts b/integration/expo-native/src/core/evidence.ts new file mode 100644 index 00000000000..6147aeb2f62 --- /dev/null +++ b/integration/expo-native/src/core/evidence.ts @@ -0,0 +1,80 @@ +import { existsSync, readFileSync, readdirSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { SHORTEST_SECRET, holdsJwt, usedSecretValues } from '../../specs/support/secret.ts'; +import { count } from './state.ts'; +import { VerifyFailure, type Brand, type EvidencePath, type EvidenceRecord } from './types.ts'; + +function files(dir: string): string[] { + return readdirSync(dir, { withFileTypes: true }).flatMap((entry) => { + const path = join(dir, entry.name); + return entry.isDirectory() ? files(path) : entry.isFile() ? [path] : []; + }); +} + +const DOUBLE_QUOTE = 0x22; +const SINGLE_QUOTE = 0x27; +const BACKSLASH = 0x5c; + +function charactersQuotedAlone(bytes: Buffer): Buffer { + const alone = Buffer.allocUnsafe(bytes.length >> 1); + let found = 0; + for (let at = 0; at + 2 < bytes.length; at += 1) { + const quote = bytes[at]; + if (quote !== DOUBLE_QUOTE && quote !== SINGLE_QUOTE) continue; + if (bytes[at + 2] === quote || (bytes[at + 2] === BACKSLASH && bytes[at + 3] === quote)) alone[found++] = bytes[at + 1]!; + } + return alone.subarray(0, found); +} + +function holdsSecret(bytes: Buffer, needles: readonly Buffer[]): boolean { + if (needles.some((needle) => bytes.includes(needle)) || holdsJwt(bytes)) return true; + const spelled = charactersQuotedAlone(bytes); + return needles.some((needle) => spelled.includes(needle)); +} + +export function sealEvidence( + dir: EvidencePath, + partial: Omit, + secretValues: readonly string[] = usedSecretValues(), +): EvidenceRecord { + const needles = secretValues.filter((v) => v.length >= SHORTEST_SECRET).map((v) => Buffer.from(v)); + if (holdsSecret(Buffer.from(JSON.stringify(partial)), needles)) { + throw new VerifyFailure('EVIDENCE_UNSAFE', 'the run record itself holds a secret value', 'report this as a verify bug; do not attach the run'); + } + const tainted = files(dir) + .filter((file) => holdsSecret(readFileSync(file), needles)) + .sort() as EvidencePath[]; + const sealed: EvidenceRecord = { ...partial, tainted, sealed: true }; + writeFileSync(join(dir, 'run.json'), `${JSON.stringify(sealed, null, 2)}\n`); + return sealed; +} + +export function readRecord(dir: EvidencePath): EvidenceRecord { + const file = join(dir, 'run.json'); + if (!existsSync(file)) throw new VerifyFailure('EVIDENCE_UNSAFE', `${dir} has no run.json, so it was never sealed`, 'run the specs again with `{cli} run`'); + return JSON.parse(readFileSync(file, 'utf8')) as EvidenceRecord; +} + +export function loggedUserIds(dir: EvidencePath): readonly string[] { + const file = join(dir, 'app.log'); + if (!existsSync(file)) return []; + return [...new Set([...readFileSync(file, 'utf8').matchAll(/"userId"\s*:\s*"([^"]+)"/g)].map((match) => match[1]!))]; +} + +export type Publishable = Brand; + +export function assertPublishable(record: EvidenceRecord, loggedUsers: readonly string[]): Publishable { + const refuse = (message: string, fix: string): never => { + throw new VerifyFailure('EVIDENCE_UNSAFE', message, fix); + }; + if (record.tainted.length > 0) refuse(`run ${record.run} has secret values in ${record.tainted.join(', ')}`, 'do not attach this run; rerun and attach the new run'); + const failed = record.results.filter((r) => r.status === 'failed' || r.status === 'interrupted'); + if (failed.length > 0) refuse(`run ${record.run} has ${failed.length} failing spec(s)`, 'fix the failures and attach a passing run'); + const incomplete = record.settings.filter((group) => group.held === false || group.e2eReport === null); + if (incomplete.length > 0) refuse(`run ${record.run} has ${count(incomplete.length, 'group')} that did not run in full on its settings: ${incomplete.map((group) => group.label).join('; ')}`, 'attach a run in which every group ran on the settings it declares'); + if (!record.results.some((r) => r.status === 'passed' || r.status === 'flaky')) refuse(`run ${record.run} passed no specs`, 'attach a run whose specs ran and passed'); + const own = new Set(record.identities.flatMap((i) => (i.userId === null ? [] : [i.userId]))); + const foreign = loggedUsers.filter((userId) => !own.has(userId)); + if (foreign.length > 0) refuse(`the app log of run ${record.run} names user(s) the run did not create: ${foreign.join(', ')}`, 'sign in only users from host.seedUser or host.newEmail'); + return record as Publishable; +} diff --git a/integration/expo-native/src/core/exec.ts b/integration/expo-native/src/core/exec.ts new file mode 100644 index 00000000000..a726c1788c1 --- /dev/null +++ b/integration/expo-native/src/core/exec.ts @@ -0,0 +1,79 @@ +import { execFileSync, spawn } from 'node:child_process'; +import type { ProcessRef } from './types.ts'; + +export interface ExecResult { + readonly code: number; + readonly stdout: string; + readonly stderr: string; +} + +export interface ExecOptions { + readonly cwd?: string; + readonly env?: Readonly>; + readonly input?: string; + readonly timeoutMs?: number; +} + +export interface CommandLine { + readonly command: string; + readonly args: readonly string[]; + readonly cwd?: string; +} + +export type Runner = (command: string, args: readonly string[], options?: ExecOptions) => Promise; + +export const run: Runner = (command, args, options = {}) => + new Promise((resolve) => { + const child = spawn(command, [...args], { + cwd: options.cwd, + env: options.env === undefined ? process.env : { ...options.env }, + stdio: [options.input === undefined ? 'ignore' : 'pipe', 'pipe', 'pipe'], + }); + let stdout = ''; + let stderr = ''; + child.stdout?.on('data', (chunk: Buffer) => (stdout += chunk.toString())); + child.stderr?.on('data', (chunk: Buffer) => (stderr += chunk.toString())); + let timedOut = false; + const timer = options.timeoutMs === undefined ? undefined : setTimeout(() => { + timedOut = true; + child.kill('SIGTERM'); + }, options.timeoutMs); + child.on('error', (error) => { + clearTimeout(timer); + resolve({ code: 127, stdout, stderr: stderr + error.message }); + }); + child.on('close', (code) => { + clearTimeout(timer); + resolve({ code: timedOut ? 124 : (code ?? 1), stdout, stderr }); + }); + if (options.input !== undefined) child.stdin?.end(options.input); + }); + +export function isAlive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch (error) { + return (error as NodeJS.ErrnoException).code === 'EPERM'; + } +} + +export const sleep = (ms: number): Promise => new Promise((resolve) => setTimeout(resolve, ms)); + +export type { ProcessRef }; + +export function currentProcess(): ProcessRef { + return { pid: process.pid, startedAt: Date.now() - process.uptime() * 1000 }; +} + +const PS_WHOLE_SECONDS_SLACK_MS = 3000; + +export function isRunning(ref: ProcessRef): boolean { + if (!isAlive(ref.pid)) return false; + try { + const started = Date.parse(execFileSync('ps', ['-o', 'lstart=', '-p', String(ref.pid)], { encoding: 'utf8' }).trim()); + return Number.isNaN(started) || Math.abs(started - ref.startedAt) < PS_WHOLE_SECONDS_SLACK_MS; + } catch { + return false; + } +} diff --git a/integration/expo-native/src/core/github-report.ts b/integration/expo-native/src/core/github-report.ts new file mode 100644 index 00000000000..2d17f8f65e1 --- /dev/null +++ b/integration/expo-native/src/core/github-report.ts @@ -0,0 +1,73 @@ +import { existsSync, readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import type { FinishedRun, Report } from 'e2e'; +import { github } from '@e2e-dev/github'; +import { protect, redact } from '../../specs/support/secret.ts'; +import type { EvidenceRecord, VerifyFailure } from './types.ts'; + +const POST_TIMEOUT_MS = 60_000; + +export function protectGitHubTokens(env: Readonly>): void { + for (const name of ['GITHUB_TOKEN', 'GH_TOKEN']) { + const value = env[name]?.trim(); + if (value) protect(value); + } +} + +type RunError = Report['run']['errors'][number]; + +export function mergeGroupReports(reports: readonly [Report, ...Report[]], notRun: readonly RunError[]): Report { + const results = new Map(); + for (const report of reports) { + for (const result of report.run.results) { + const seen = results.get(result.id); + if (seen === undefined || (result.selected && !seen.selected)) results.set(result.id, result); + } + } + const first = reports[0]; + const worst = reports.find((report) => report.run.status !== 'passed') ?? first; + return { + ...first, + run: { + ...first.run, + status: notRun.length > 0 && worst.run.status === 'passed' ? 'failed' : worst.run.status, + exitCode: notRun.length > 0 && worst.run.exitCode === 0 ? 1 : worst.run.exitCode, + finishedAt: reports.at(-1)!.run.finishedAt, + serialGroups: reports.flatMap((report) => report.run.serialGroups), + results: [...results.values()], + errors: [...reports.flatMap((report) => report.run.errors), ...notRun], + }, + }; +} + +export interface GitHubReportInput { + readonly record: EvidenceRecord; + readonly failures: readonly { readonly label: string; readonly failure: VerifyFailure }[]; + readonly packageDir: string; +} + +export async function reportToGitHub({ record, failures, packageDir }: GitHubReportInput): Promise { + if (record.tainted.length > 0) return ['not reported: a file of this run holds a secret value']; + try { + const files = record.settings.flatMap((group) => (group.e2eReport !== null && existsSync(group.e2eReport) ? [group.e2eReport] : [])); + const [first, ...rest] = files.map((file) => JSON.parse(readFileSync(file, 'utf8')) as Report); + if (first === undefined) return ['not reported: no group of this run wrote a report']; + const report = mergeGroupReports( + [first, ...rest], + failures.map(({ label, failure }) => ({ category: 'infrastructure', code: failure.code, message: redact(`${label}: ${failure.message}`), retryable: false })), + ); + const run: FinishedRun = { + report, + status: report.run.status, + exitCode: report.run.exitCode, + projectRoot: packageDir, + reportPath: files[0], + artifactsRoot: join(dirname(files[0]!), 'artifacts'), + aiTracePath: undefined, + }; + const rows = await github({ key: record.platform }).onRunFinished!(run, AbortSignal.timeout(POST_TIMEOUT_MS)); + return (rows ?? []).map((row) => row.text); + } catch (error) { + return [redact(`not reported: ${(error as Error).message ?? String(error)}`)]; + } +} diff --git a/integration/expo-native/src/core/keys.ts b/integration/expo-native/src/core/keys.ts new file mode 100644 index 00000000000..56aa7553f56 --- /dev/null +++ b/integration/expo-native/src/core/keys.ts @@ -0,0 +1,12 @@ +import { AGENT_CREDENTIAL_VARIABLES, INSTANCE_SECRET_VARIABLES, PLATFORM_CREDENTIAL_VARIABLES } from './launch.mjs'; +import type { Secret } from '../../specs/support/secret.ts'; +import type { PublishableKey } from './types.ts'; + +export interface InstanceKeys { + readonly pk: PublishableKey; + readonly sk: Secret<'clerk-secret-key'>; +} + +export function withoutClerkKeys>>(env: T): T { + return Object.fromEntries(Object.entries(env).filter(([name]) => !PLATFORM_CREDENTIAL_VARIABLES.includes(name) && !AGENT_CREDENTIAL_VARIABLES.includes(name) && !INSTANCE_SECRET_VARIABLES.includes(name))) as T; +} diff --git a/integration/expo-native/src/core/launch.d.mts b/integration/expo-native/src/core/launch.d.mts new file mode 100644 index 00000000000..2b51fed2c67 --- /dev/null +++ b/integration/expo-native/src/core/launch.d.mts @@ -0,0 +1,5 @@ +export const PLATFORM_CREDENTIAL_VARIABLES: readonly string[]; +export const AGENT_CREDENTIAL_VARIABLES: readonly string[]; +export const INSTANCE_SECRET_VARIABLES: readonly string[]; +export function supportsNode(version: string): boolean; +export function ensureRuntime(): Promise; diff --git a/integration/expo-native/src/core/launch.mjs b/integration/expo-native/src/core/launch.mjs new file mode 100644 index 00000000000..448f47970c4 --- /dev/null +++ b/integration/expo-native/src/core/launch.mjs @@ -0,0 +1,37 @@ +import { spawn, spawnSync } from 'node:child_process'; +import { existsSync } from 'node:fs'; +import { delimiter, dirname } from 'node:path'; + +export const PLATFORM_CREDENTIAL_VARIABLES = ['CLERK_PLATFORM_API_KEY', 'CLERK_PLATFORM_API_KEY_FILE', 'VERIFY_PLATFORM_KEY_REFERENCE']; +export const AGENT_CREDENTIAL_VARIABLES = ['AI_GATEWAY_API_KEY', 'AI_GATEWAY_API_KEY_FILE']; +export const INSTANCE_SECRET_VARIABLES = ['CLERK_SECRET_KEY']; + +export function supportsNode(version) { + const [major, minor] = version.split('.').map(Number); + return major === 24 && minor >= 8; +} + +function rerun(command, args, env) { + return new Promise((resolve) => { + const child = spawn(command, args, { stdio: 'inherit', env }); + for (const signal of ['SIGINT', 'SIGTERM', 'SIGHUP']) process.on(signal, () => child.kill(signal)); + child.on('error', () => resolve(127)); + child.on('close', (code, signal) => resolve(code ?? (signal === null ? 1 : 130))); + }); +} + +export async function ensureRuntime() { + if (!supportsNode(process.versions.node)) { + const withoutKey = Object.fromEntries(Object.entries(process.env).filter(([name]) => !PLATFORM_CREDENTIAL_VARIABLES.includes(name) && !AGENT_CREDENTIAL_VARIABLES.includes(name) && !INSTANCE_SECRET_VARIABLES.includes(name))); + const node24 = process.env.VERIFY_NODE_RERUN === undefined ? (spawnSync('npx', ['-y', 'node@24', '-p', 'process.execPath'], { encoding: 'utf8', env: withoutKey }).stdout ?? '').trim() : ''; + if (node24 !== '' && existsSync(node24)) { + console.error(`note this is Node ${process.versions.node}; running under node@24 through npx`); + process.exit(await rerun(node24, process.argv.slice(1), { ...process.env, PATH: `${dirname(node24)}${delimiter}${process.env.PATH ?? ''}`, VERIFY_NODE_RERUN: '1' })); + } + const message = `this CLI needs Node 24.8.0 or newer on 24 and this is Node ${process.versions.node}; npx could not fetch node@24`; + const fix = 'install Node 24.8.0 or newer on 24 (nvm install 24 && nvm use 24) and rerun'; + if (process.argv.includes('--json')) console.log(JSON.stringify({ ok: false, error: { code: 'NOT_READY', message, fix, retryable: false } })); + else console.error(`FAIL node ${message}\n fix: ${fix}`); + process.exit(3); + } +} diff --git a/integration/expo-native/src/core/ledgers.ts b/integration/expo-native/src/core/ledgers.ts new file mode 100644 index 00000000000..085d1a5917a --- /dev/null +++ b/integration/expo-native/src/core/ledgers.ts @@ -0,0 +1,81 @@ +import { existsSync, readFileSync, readdirSync } from 'node:fs'; +import { join } from 'node:path'; +import type { Instances } from './instances/instances.ts'; +import { count } from './state.ts'; +import { isRunning } from './exec.ts'; +import { newEntryId, openWorkspace, type Workspace } from './workspace.ts'; +import type { LedgerEntry, Platform } from './types.ts'; + +export type ProcessEntry = Extract; + +export const openProcesses = (workspace: Workspace): readonly ProcessEntry[] => workspace.unclosedEntries().filter((entry): entry is ProcessEntry => entry.kind === 'process'); + +export const processesIn = (workspace: Workspace, scope: { readonly platforms: readonly Platform[]; readonly sharedByEveryLease: boolean }): readonly ProcessEntry[] => + openProcesses(workspace).filter((entry) => (entry.platform === undefined ? scope.sharedByEveryLease : scope.platforms.includes(entry.platform))); + +export function stopProcesses(workspace: Workspace, entries: readonly ProcessEntry[]): readonly string[] { + const stopped: string[] = []; + for (const entry of entries) { + if (isRunning({ pid: entry.pid, startedAt: Date.parse(entry.startedAt) })) { + try { + process.kill(entry.pid, entry.what === 'recorder' ? 'SIGINT' : 'SIGTERM'); + stopped.push(`${entry.what} ${entry.pid}`); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ESRCH') throw error; + stopped.push(`${entry.what} ${entry.pid} had already exited`); + } + } else { + stopped.push(`${entry.what} ${entry.pid} had already exited`); + } + workspace.append({ id: newEntryId(), kind: 'done', ref: entry.id }); + } + return stopped; +} + +export interface DaemonInfo { + readonly pid: number; + readonly startedAt: number; +} + +export function readDaemonInfo(stateDir: string): DaemonInfo | null { + const file = join(stateDir, 'daemon.json'); + if (!existsSync(file)) return null; + try { + const raw = JSON.parse(readFileSync(file, 'utf8')) as { pid?: unknown; processStartTime?: unknown }; + if (typeof raw.pid !== 'number' || typeof raw.processStartTime !== 'string') return null; + return { pid: raw.pid, startedAt: Date.parse(raw.processStartTime) }; + } catch { + return null; + } +} + +export function ledgerAgentDeviceDaemon(workspace: Workspace): void { + const daemon = readDaemonInfo(workspace.agentDeviceDir); + if (daemon === null || !isRunning(daemon)) return; + const known = workspace.unclosedEntries().some((e) => e.kind === 'process' && e.what === 'agent-device' && e.pid === daemon.pid); + if (!known) workspace.append({ id: newEntryId(), kind: 'process', what: 'agent-device', pid: daemon.pid, startedAt: new Date(daemon.startedAt).toISOString() }); +} + +export async function finishOrphanLedgers( + home: string, + self: string, + instances: Instances, + progress: (line: string) => void, +): Promise { + const dir = join(home, 'ledgers'); + if (!existsSync(dir)) return; + for (const name of readdirSync(dir).filter((n) => n.endsWith('.owner'))) { + const [worktree = '', packageDir] = readFileSync(join(dir, name), 'utf8').trim().split('\n'); + if (worktree === self || existsSync(worktree)) continue; + const ledger = openWorkspace({ packageDir: packageDir ?? worktree, worktree, home }); + if (ledger.unclosedEntries().length === 0) continue; + try { + const stopped = stopProcesses(ledger, openProcesses(ledger)); + const deleted = await instances.finish(ledger, { keepApplications: false }, progress); + for (const entry of ledger.unclosedEntries()) ledger.append({ id: newEntryId(), kind: 'done', ref: entry.id }); + progress(`reap ledger of ${worktree} (worktree is gone) deleted ${count(deleted.length, 'application')}, stopped ${stopped.join(', ') || 'nothing'}`); + } catch (error) { + progress(`reap ledger of ${worktree} left open: ${(error as Error).message}`); + } + } +} diff --git a/integration/expo-native/src/core/manifest.ts b/integration/expo-native/src/core/manifest.ts new file mode 100644 index 00000000000..7ee6f652a28 --- /dev/null +++ b/integration/expo-native/src/core/manifest.ts @@ -0,0 +1,42 @@ +import { createHash } from 'node:crypto'; +import { readFileSync, readdirSync, statSync, writeFileSync } from 'node:fs'; +import { dirname, join, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const CORE_DIR = dirname(fileURLToPath(import.meta.url)); +const PACKAGE_DIR = join(CORE_DIR, '..', '..'); +const MANIFEST_FILE = join(CORE_DIR, 'MANIFEST'); + +export const SHARED_ROOTS = ['src/core', 'specs/support', 'specs/fixtures.ts', 'e2e.config.ts'] as const; + +function filesUnder(path: string): string[] { + const found = statSync(path, { throwIfNoEntry: false }); + if (found === undefined) return []; + if (found.isFile()) return path === MANIFEST_FILE ? [] : [path]; + return readdirSync(path).flatMap((name) => filesUnder(join(path, name))); +} + +function computeManifest(): string { + return SHARED_ROOTS.flatMap((root) => filesUnder(join(PACKAGE_DIR, root))) + .map((file) => `${createHash('sha256').update(readFileSync(file)).digest('hex')} ${relative(PACKAGE_DIR, file).split('\\').join('/')}`) + .sort((a, b) => a.slice(66).localeCompare(b.slice(66))) + .join('\n') + .concat('\n'); +} + +export function manifestDrift(): readonly string[] { + const parse = (text: string) => new Map(text.split('\n').filter(Boolean).map((line) => [line.slice(66), line.slice(0, 64)] as const)); + let committed: Map; + try { + committed = parse(readFileSync(MANIFEST_FILE, 'utf8')); + } catch { + return ['MANIFEST']; + } + const actual = parse(computeManifest()); + const names = new Set([...committed.keys(), ...actual.keys()]); + return [...names].filter((name) => committed.get(name) !== actual.get(name)).sort(); +} + +if (import.meta.main && process.argv.includes('--write')) { + writeFileSync(MANIFEST_FILE, computeManifest()); +} diff --git a/integration/expo-native/src/core/publish.ts b/integration/expo-native/src/core/publish.ts new file mode 100644 index 00000000000..40715230393 --- /dev/null +++ b/integration/expo-native/src/core/publish.ts @@ -0,0 +1,75 @@ +import { existsSync, readFileSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { run as defaultRunner, type Runner } from './exec.ts'; +import type { Publishable } from './evidence.ts'; +import { VerifyFailure, type AttachResult, type EvidencePath, type HostAdapter } from './types.ts'; + +interface Posted { + readonly pr: number; + readonly commentUrl: string; + readonly posted: readonly EvidencePath[]; +} + +function tally(results: Publishable['results']): string { + const flaky = results.filter((r) => r.status === 'flaky').length; + return `${results.filter((r) => r.status === 'passed').length} of ${results.length} passed${flaky === 0 ? '' : `, ${flaky} flaky (passed only on a retry)`}.`; +} + +export function commentBody(evidence: Publishable): string { + const lines = [ + `verify run \`${evidence.run}\` on ${evidence.platform} (${evidence.device}), build \`${evidence.build}\`, ${tally(evidence.results)}`, + '', + ...evidence.results.map((r) => `- ${r.status}: \`${r.spec.path}\` ${r.title}`), + ]; + for (const group of evidence.settings) { + if (group.askedBy === null) continue; + const results = evidence.results.filter((r) => group.specs.includes(r.spec.path)); + lines.push('', `Instance settings \`${group.label}\` (declared by \`${group.askedBy}\`): ${tally(results)}`); + } + return lines.join('\n'); +} + +const ATTACH_FIX = 'install a gh build whose `gh pr comment` has --attach'; + +export async function missingAttach(runner: Runner): Promise<{ readonly why: string; readonly fix: string } | null> { + const help = await runner('gh', ['pr', 'comment', '--help']); + if (help.code === 0 && help.stdout.includes('--attach')) return null; + return { why: help.code === 0 ? 'this gh has no `gh pr comment --attach`' : 'gh is not installed', fix: ATTACH_FIX }; +} + +export async function postToPullRequest( + evidence: Publishable, + dir: EvidencePath, + host: HostAdapter, + pr: number, + screenshots: 'all' | readonly string[], + runner: Runner = defaultRunner, +): Promise { + const postedFile = join(dir, `posted-${pr}.json`); + if (existsSync(postedFile)) { + const previous = JSON.parse(readFileSync(postedFile, 'utf8')) as Posted; + return { verb: 'attach', commentUrl: previous.commentUrl, posted: previous.posted, alreadyPosted: true }; + } + const chosen = + screenshots === 'all' + ? evidence.screenshots + : screenshots.map((label) => { + const shot = evidence.screenshots.find((s) => s.label === label); + if (shot === undefined) { + throw new VerifyFailure('USAGE', `run ${evidence.run} has no screenshot labelled ${label}`, `use one of: ${evidence.screenshots.map((s) => s.label).join(', ') || '(none)'}`); + } + return shot; + }); + const files = [...evidence.videos, ...chosen.map((s) => s.path)]; + const missing = files.length === 0 ? null : await missingAttach(runner); + if (missing !== null) throw new VerifyFailure('NOT_READY', `${missing.why}, so the video and screenshots of run ${evidence.run} cannot be posted`, missing.fix); + const args = ['pr', 'comment', String(pr), '--repo', host.githubRepo, '--body', commentBody(evidence), ...files.flatMap((f) => ['--attach', f])]; + const result = await runner('gh', args); + if (result.code !== 0) { + throw new VerifyFailure('NOT_READY', `gh pr comment failed: ${result.stderr.trim()}`, 'check `gh auth status` and that the PR exists'); + } + const commentUrl = /https:\/\/github\.com\/\S+/.exec(result.stdout)?.[0] ?? result.stdout.trim(); + const posted: Posted = { pr, commentUrl, posted: files }; + writeFileSync(postedFile, `${JSON.stringify(posted, null, 2)}\n`); + return { verb: 'attach', commentUrl, posted: files, alreadyPosted: false }; +} diff --git a/integration/expo-native/src/core/slot.ts b/integration/expo-native/src/core/slot.ts new file mode 100644 index 00000000000..68a89dcaf7f --- /dev/null +++ b/integration/expo-native/src/core/slot.ts @@ -0,0 +1,73 @@ +import { randomUUID } from 'node:crypto'; +import { linkSync, mkdirSync, readFileSync, readdirSync, rmSync, statSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; + +const NAME = /^\d{12}$/; +const FREE = 'free'; +const STALE_WRITER_WINDOW_MS = 60_000; + +const nameOf = (gen: number) => String(gen).padStart(12, '0'); + +function generations(dir: string): number[] { + let names: string[]; + try { + names = readdirSync(dir); + } catch { + return []; + } + return names.filter((n) => NAME.test(n)).map(Number).sort((a, b) => a - b); +} + +export interface SlotState { + readonly gen: number; + readonly value: string | null; +} + +export function readSlot(dir: string): SlotState { + for (;;) { + const gens = generations(dir); + const gen = gens.at(-1); + if (gen === undefined) return { gen: 0, value: null }; + try { + const text = readFileSync(join(dir, nameOf(gen)), 'utf8'); + return { gen, value: text === FREE ? null : text }; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error; + } + } +} + +function linkedIntoPrunedGap(gens: readonly number[], linked: number): boolean { + return gens.some((g) => g > linked); +} + +export function compareAndSwapSlot(dir: string, from: number, value: string | null): boolean { + mkdirSync(dir, { recursive: true }); + const next = from + 1; + const file = join(dir, nameOf(next)); + const staged = join(dir, `.${randomUUID()}`); + writeFileSync(staged, value ?? FREE, { mode: 0o600 }); + try { + linkSync(staged, file); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'EEXIST') return false; + throw error; + } finally { + rmSync(staged, { force: true }); + } + const gens = generations(dir); + if (linkedIntoPrunedGap(gens, next)) { + rmSync(file, { force: true }); + return false; + } + const cutoff = Date.now() - STALE_WRITER_WINDOW_MS; + for (const g of gens) { + if (g >= next - 1) continue; + try { + if (statSync(join(dir, nameOf(g))).mtimeMs < cutoff) rmSync(join(dir, nameOf(g)), { force: true }); + } catch { + continue; + } + } + return true; +} diff --git a/integration/expo-native/src/core/state.ts b/integration/expo-native/src/core/state.ts new file mode 100644 index 00000000000..fd56280511e --- /dev/null +++ b/integration/expo-native/src/core/state.ts @@ -0,0 +1 @@ +export const count = (n: number, noun: string): string => `${n} ${noun}${n === 1 ? '' : 's'}`; diff --git a/integration/expo-native/src/core/types.ts b/integration/expo-native/src/core/types.ts new file mode 100644 index 00000000000..d580634e83c --- /dev/null +++ b/integration/expo-native/src/core/types.ts @@ -0,0 +1,385 @@ +import type { Brand, Platform, RunId, TestEmail } from '../../specs/support/types.ts'; + +export type { + AppEntry, + AppLocators, + AuthMode, + Brand, + HostFixture, + HostLaunch, + LaunchId, + LaunchOptions, + Platform, + PublishableKey, + RunId, + SecretLike, + SecretSink, + SeedOptions, + SeededUser, + StorageScope, + TestEmail, + TestPhone, +} from '../../specs/support/types.ts'; + +export const CLI_PLACEHOLDER = '{cli}'; + +export type AcquireLock = Brand<{ readonly platform: Platform }, 'AcquireLock'>; + +export type BuildKey = Brand; +export type FeatureName = Brand; +export type EvidencePath = Brand; +export type ScratchPath = Brand; + +export type Json = null | boolean | number | string | readonly Json[] | { readonly [key: string]: Json }; + +export interface InstanceSettings { + readonly config: { readonly [key: string]: Json }; + readonly environment: { readonly [leaf: string]: Json }; +} + +export interface InstanceView { + readonly id: string; + readonly name: string; + readonly created: boolean; + readonly settings: string; +} + +export interface ApplicationView { + readonly name: string; +} + +export type BackendKind = 'local'; + +export type SpecSelection = { readonly all: true } | { readonly selectors: readonly string[] }; + +export type Command = + | { readonly verb: 'doctor'; readonly platform?: Platform; readonly backend?: BackendKind; readonly live: boolean } + | { readonly verb: 'up'; readonly platform?: Platform; readonly backend?: BackendKind; readonly waitSeconds: number } + | { + readonly verb: 'run'; + readonly selection: SpecSelection; + readonly platform?: Platform; + readonly backend?: BackendKind; + readonly grep?: string; + readonly video: boolean; + readonly retries: number; + readonly githubReport: boolean; + readonly waitSeconds: number; + } + | { readonly verb: 'screen'; readonly platform?: Platform; readonly png: boolean } + | { readonly verb: 'attach'; readonly run: RunId; readonly pr: number; readonly screenshots: 'all' | readonly string[] } + | { readonly verb: 'down'; readonly platform?: Platform; readonly stale: boolean; readonly dryRun: boolean }; + +export type Verb = Command['verb']; +export type RunCommand = Extract; + +export interface Invocation { + readonly command: Command; + readonly json: boolean; +} + +export type ErrorCode = + | 'USAGE' + | 'NOT_READY' + | 'POOL_FULL' + | 'LEASE_LOST' + | 'DEVICE_BUSY' + | 'BUILD_FAILED' + | 'KEYS_MISSING' + | 'INSTANCE_MISCONFIGURED' + | 'NOT_TEST_IDENTITY' + | 'NO_SPECS' + | 'E2E_CRASHED' + | 'EVIDENCE_UNSAFE' + | 'UNSUPPORTED' + | 'RATE_LIMITED'; + +export const RETRYABLE: ReadonlySet = new Set(['POOL_FULL', 'DEVICE_BUSY', 'LEASE_LOST', 'RATE_LIMITED']); + +export class VerifyFailure extends Error { + readonly code: ErrorCode; + readonly fix: string; + constructor(code: ErrorCode, message: string, fix: string) { + super(message); + this.code = code; + this.fix = fix; + } +} + +export type DoctorCheckId = + | 'node' | 'xcode' | 'jdk' | 'e2e-pins' | 'template' | 'proxy-trust' + | 'settings' | 'build' | 'gh-attach' | 'core-drift' | 'stale-claims' | 'lane-ports' + | 'instances' | 'clerk-api' | 'agent' + | 'backend' + | `live-${string}`; + +interface DoctorCheckBase { + readonly id: DoctorCheckId; + readonly detail: string; + readonly fix?: string; +} +export type DoctorCheck = + | (DoctorCheckBase & { readonly ok: boolean; readonly state?: undefined }) + | (DoctorCheckBase & { readonly ok: true; readonly state: 'warning' | 'not-run' }); + +export interface DoctorReport { + readonly verb: 'doctor'; + readonly ok: boolean; + readonly backend: Readonly>>; + readonly checks: readonly DoctorCheck[]; +} + +export interface LeaseView { + readonly platform: Platform; + readonly backend: BackendKind; + readonly device: string; + readonly installedBuild: BuildKey | null; + readonly expiresAt: string | null; + readonly renewed: boolean; +} + +export interface BuildView { + readonly platform: Platform; + readonly key: BuildKey; + readonly source: BuildSource; + readonly reused: boolean; + readonly seconds: number; +} + +export interface UpResult { + readonly verb: 'up'; + readonly leases: readonly LeaseView[]; + readonly builds: readonly BuildView[]; + readonly instances: readonly InstanceView[]; +} + +export interface RunResult { + readonly verb: 'run'; + readonly dir: EvidencePath; + readonly record: EvidenceRecord; + readonly next: string; +} + +export interface ScreenNode { + readonly role: string; + readonly name: string | null; + readonly testId: string | null; + readonly text: string | null; + readonly depth: number; + readonly locator: string | null; +} + +export interface ScreenResult { + readonly verb: 'screen'; + readonly platform: Platform; + readonly device: string; + readonly nodes: readonly ScreenNode[]; + readonly png: ScratchPath | null; +} + +export interface AttachResult { + readonly verb: 'attach'; + readonly commentUrl: string; + readonly posted: readonly EvidencePath[]; + readonly alreadyPosted: boolean; +} + +export type DownResult = + | { + readonly verb: 'down'; + readonly dryRun: false; + readonly released: readonly LeaseView[]; + readonly deletedApplications: readonly ApplicationView[]; + readonly stoppedProcesses: readonly string[]; + readonly keptRuns: readonly RunId[]; + } + | { + readonly verb: 'down'; + readonly dryRun: true; + readonly wouldRelease: readonly LeaseView[]; + readonly wouldDelete: readonly DeletionTarget[]; + readonly wouldStop: readonly string[]; + readonly keptRuns: readonly RunId[]; + }; + +export interface DeletionTarget { + readonly kind: 'application'; + readonly name: string; +} + +export type VerbResult = DoctorReport | UpResult | RunResult | ScreenResult | AttachResult | DownResult; + +export type SpecKind = 'golden' | 'explored'; +export interface SpecRef { + readonly kind: SpecKind; + readonly path: string; + readonly feature: FeatureName | null; +} + +interface BuiltAppBase { + readonly platform: Platform; + readonly key: BuildKey; + readonly appId: string; +} +export interface LocalBuild extends BuiltAppBase { + readonly source: 'local'; + readonly path: ScratchPath; +} +export type BuiltApp = LocalBuild; +export type BuildSource = BuiltApp['source']; + +export type DeviceName = `verify-${Platform}-${number}`; + +interface LeaseBase { + readonly platform: Platform; + readonly acquiredAt: string; + readonly installedBuild: BuildKey | null; +} +export interface LocalLease extends LeaseBase { + readonly backend: 'local'; + readonly slot: number; + readonly deviceName: DeviceName; + readonly deviceId: string; + readonly claimNonce: string; +} +export type Lease = LocalLease; + +export type LedgerEntry = + | { readonly id: string; readonly kind: 'lease-intent'; readonly platform: Platform; readonly backend: BackendKind; readonly worktree: string } + | { readonly id: string; readonly kind: 'lease-held'; readonly platform: Platform; readonly backend: BackendKind; readonly sessionId: string | null; readonly deviceId: string | null } + | { readonly id: string; readonly kind: 'application'; readonly name: string; readonly workspace: string } + | { readonly id: string; readonly kind: 'user'; readonly run: RunId; readonly userId: string; readonly email: TestEmail } + | { readonly id: string; readonly kind: 'process'; readonly what: 'metro' | 'watch' | 'recorder' | 'agent-device'; readonly pid: number; readonly startedAt: string; readonly platform?: Platform } + | { readonly id: string; readonly kind: 'done'; readonly ref: string }; + +export type SpecStatus = 'passed' | 'failed' | 'skipped' | 'flaky' | 'interrupted'; +export interface SpecResult { + readonly spec: SpecRef; + readonly title: string; + readonly platform: Platform; + readonly status: SpecStatus; + readonly seconds: number; + readonly attempts: number; + readonly error: string | null; + readonly skipReason: string | null; + readonly skippedBy: 'platform' | null; + readonly tags: readonly string[]; + readonly failurePage: EvidencePath | null; + readonly failureScreen: EvidencePath | null; + readonly failureScreenshot: EvidencePath | null; +} + +export interface EvidenceRecord { + readonly run: RunId; + readonly startedAt: string; + readonly finishedAt: string; + readonly repo: HostAdapter['repo']; + readonly gitHead: string; + readonly dirty: boolean; + readonly platform: Platform; + readonly backend: BackendKind; + readonly device: string; + readonly build: BuildKey; + readonly results: readonly SpecResult[]; + readonly videos: readonly EvidencePath[]; + readonly screenshots: readonly { readonly label: string; readonly path: EvidencePath }[]; + readonly appLog: EvidencePath | null; + readonly e2eReport: EvidencePath; + readonly identities: readonly { readonly email: TestEmail; readonly userId: string | null }[]; + readonly settings: readonly { + readonly label: string; + readonly askedBy: string | null; + readonly specs: readonly string[]; + readonly application: string | null; + readonly changed: boolean; + readonly held: boolean; + readonly e2eReport: EvidencePath | null; + }[]; + readonly tainted: readonly EvidencePath[]; + readonly sealed: true; +} + +export interface RuntimeProcess { + readonly what: 'metro' | 'watch'; + readonly pid: number; + readonly startedAt: number; + readonly platform?: Platform; +} + +export interface HostRuntime { + readonly devServer: string | null; + readonly processes: readonly RuntimeProcess[]; +} + +export interface E2EInvocation { + readonly args: readonly string[]; + readonly env: Readonly>; +} + +export interface DeviceWait { + readonly seconds: number; + readonly busyFix: string; + readonly onWait?: (owner: ProcessRef) => void; +} + +export interface AcquireRequest { + readonly platform: Platform; + readonly worktree: string; + readonly waitSeconds: number; + readonly app: BuiltApp; + readonly retryWith: string; + readonly progress: (line: string) => void; +} + +export interface ProcessRef { + readonly pid: number; + readonly startedAt: number; +} + +export interface Recording { + readonly process: ProcessRef | null; + stop(): Promise; +} + +export interface Availability { + readonly usable: boolean; + readonly why: string; + readonly fix?: string; +} + +export interface DoctorOptions { + readonly live: boolean; + readonly worktree: string; + readonly progress: (line: string) => void; +} + +export interface DeviceBackend { + readonly kind: BackendKind; + readonly platform: Platform; + availability(): Availability; + acquire(request: AcquireRequest): Promise; + check(lease: L): Promise<'held' | 'lost'>; + install(lease: L, app: BuiltApp, progress: (line: string) => void): Promise; + release(lease: L): Promise; + reapable(owner?: string): Promise; + startRecording(lease: L, into: EvidencePath): Promise; + logs(lease: L, since: Date, extraPredicate?: string): Promise; + describe(lease: L): string; + readonly requirement: string; + doctorChecks(options: DoctorOptions): Promise<{ readonly toolchain: readonly DoctorCheck[]; readonly device: readonly DoctorCheck[] }>; +} + +export const LOCAL_POOL: Readonly> = { ios: 4, android: 2 }; + +export interface HostAdapter { + readonly repo: 'clerk-ios' | 'clerk-android' | 'clerk-expo'; + readonly cli: string; + readonly platforms: readonly Platform[]; + readonly githubRepo: string; + appId(platform: Platform): string; + buildInputs(platform: Platform, backend: BackendKind): readonly string[]; + build(platform: Platform, key: BuildKey, into: ScratchPath, progress: (line: string) => void): Promise; + readonly logPredicates?: Readonly>>; + readonly backends: readonly DeviceBackend[]; + runtime?(lease: Lease, progress: (line: string) => void): Promise; +} diff --git a/integration/expo-native/src/core/verbs.ts b/integration/expo-native/src/core/verbs.ts new file mode 100644 index 00000000000..330b90b3ea4 --- /dev/null +++ b/integration/expo-native/src/core/verbs.ts @@ -0,0 +1,675 @@ +import { appendFileSync, existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { isOrphaned, readClaims } from './claims.ts'; +import { check, type AppliedInstance, type Instances } from './instances/instances.ts'; +import { STANDARD, SettingsRefused, planGroups, readSpecText, settingsFileOf, sourceHash, straySettingsFile, type SettingsGroup } from './instances/settings.ts'; +import { agentCheck, type AgentSource } from './agent.ts'; +import type { Agent } from '../../specs/support/agent.ts'; +import { runPassword } from '../../specs/support/clerk.ts'; +import { withoutClerkKeys } from './keys.ts'; +import { supportsNode } from './launch.mjs'; +import { openApplications } from './instances/throwaway.ts'; +import { agentDevice as agentDeviceOn } from '../../specs/support/device.ts'; +import type { Inputs, Target } from '../../specs/support/inputs.ts'; +import { backendFor, computeBuildKey, describeChoice, deviceOf, ensureLease, leaseLine, leaseView, readBuiltApp, releaseLease, selectBackend, type BackendChoice, type LeaseOutcome } from './devices.ts'; +import { assertSomethingRan, collectScreenshots, e2eOutputDir, invokeE2E, parseE2EReport, planE2E, resolveSpecs } from './e2e.ts'; +import { startBroker, type Broker } from './broker.ts'; +import { watchDriver, type DriverWatch } from './driver.ts'; +import { assertPublishable, loggedUserIds, readRecord, sealEvidence } from './evidence.ts'; +import { protectGitHubTokens, reportToGitHub } from './github-report.ts'; +import { isRunning, type Runner } from './exec.ts'; +import { ledgerAgentDeviceDaemon, processesIn, readDaemonInfo, stopProcesses, type ProcessEntry } from './ledgers.ts'; +import { SHARED_ROOTS, manifestDrift } from './manifest.ts'; +import { missingAttach, postToPullRequest } from './publish.ts'; +import { redact } from '../../specs/support/secret.ts'; +import { count } from './state.ts'; +import { newEntryId, parseRunId, type Workspace } from './workspace.ts'; +import { + VerifyFailure, + type AttachResult, + type BackendKind, + type Command, + type DeviceBackend, + type ProcessRef, + type Recording, + type DoctorCheck, + type DoctorReport, + type DownResult, + type EvidencePath, + type EvidenceRecord, + type HostAdapter, + type InstanceView, + type Lease, + type LeaseView, + type Platform, + type RunId, + type RunResult, + type ScratchPath, + type ScreenNode, + type ScreenResult, + type SpecRef, + type SpecResult, + type UpResult, +} from './types.ts'; + +export interface Deps { + readonly host: HostAdapter; + readonly workspace: Workspace; + readonly runner: Runner; + readonly env: Readonly>; + readonly progress: (line: string) => void; + readonly instances: Instances; + readonly agent?: AgentSource; + readonly fetch?: typeof fetch; + readonly watchDriver?: typeof watchDriver; +} + +async function leaseWithInstances(deps: Deps, instances: { readonly willChange: boolean }, lease: () => Promise): Promise<[LeaseOutcome, readonly InstanceView[]]> { + await deps.instances.access(); + if (instances.willChange) { + const ensured = await deps.instances.ensure(instances, deps.progress); + return [await lease(), ensured]; + } + const [leased, ensured] = await Promise.allSettled([lease(), deps.instances.ensure(instances, deps.progress)]); + if (leased.status === 'rejected') throw leased.reason; + if (ensured.status === 'rejected') { + const failure = ensured.reason instanceof VerifyFailure ? ensured.reason : new VerifyFailure('NOT_READY', (ensured.reason as Error).message, 'run `{cli} doctor`'); + throw new VerifyFailure(failure.code, failure.message, `${failure.fix}; the device stays leased until \`{cli} down\``); + } + return [leased.value, ensured.value]; +} + +const platformOf = (host: HostAdapter, platform: Platform | undefined): Platform => { + const chosen = platform ?? host.platforms[0]; + if (chosen === undefined || !host.platforms.includes(chosen)) { + throw new VerifyFailure('USAGE', `${host.repo} does not support ${platform}`, `use --platform ${host.platforms.join(' or ')}`); + } + return chosen; +}; + +function chooseBackend(deps: Deps, platform: Platform, command: { readonly backend?: BackendKind }): BackendChoice { + return selectBackend(deps.host, platform, command.backend, deps.workspace.readLease(platform)); +} + +function readJson(file: string): Record | null { + return existsSync(file) ? (JSON.parse(readFileSync(file, 'utf8')) as Record) : null; +} + +export async function doctor(deps: Deps, command: Extract): Promise { + const { host, workspace, runner } = deps; + const platform = platformOf(host, command.platform); + const choice = chooseBackend(deps, platform, command); + const { backend } = choice; + const packageDir = workspace.packageDir; + const checks: DoctorCheck[] = [check('backend', true, describeChoice(choice), '')]; + + const node = process.versions.node; + checks.push(check('node', supportsNode(node), node, 'install Node 24.8.0 or newer on 24 (nvm install 24)')); + const backendChecks = await backend.doctorChecks({ + live: command.live, + worktree: workspace.worktree, + progress: deps.progress, + }); + checks.push(...backendChecks.toolchain); + + const pkg = readJson(join(packageDir, 'package.json')); + const pins = (pkg?.devDependencies ?? {}) as Record; + const installed = (name: string) => readJson(join(packageDir, 'node_modules', name, 'package.json'))?.version as string | undefined; + const pinned = ['e2e', '@e2e-dev/mobile', '@e2e-dev/github', 'ai'].map((name) => ({ name, want: pins[name], have: installed(name) })); + checks.push( + check( + 'e2e-pins', + pinned.every((p) => p.want !== undefined && p.want === p.have), + pinned.map((p) => `${p.name} ${p.have ?? 'missing'}${p.have === p.want ? '' : ` (pinned ${p.want})`}`).join(', '), + `cd ${packageDir} && npm ci`, + ), + ); + checks.push(...backendChecks.device); + + checks.push(...(await deps.instances.doctorChecks({ live: command.live }, deps.progress))); + checks.push(agentCheck(deps.agent ?? (() => null))); + + const key = await computeBuildKey(host, platform, backend.kind, workspace.worktree); + const up = ['{cli} up', ...(host.platforms.length > 1 ? [`--platform ${platform}`] : []), ...(command.backend === undefined ? [] : [`--backend ${command.backend}`])].join(' '); + const built = readBuiltApp(workspace, key); + checks.push(check('build', built !== null, built === null ? `no ${host.appId(platform)} build for ${key}` : `${key} at ${built.path}`, up)); + + const noAttach = await missingAttach(runner); + checks.push( + noAttach === null + ? check('gh-attach', true, 'gh pr comment supports --attach', '') + : { id: 'gh-attach', ok: true, state: 'warning', detail: `${noAttach.why}, so \`{cli} attach\` cannot post a run's video and screenshots from this machine`, fix: noAttach.fix }, + ); + + const stale = readClaims(workspace.claimsDir, platform).filter(isOrphaned); + checks.push(check('stale-claims', stale.length === 0, stale.length === 0 ? 'none' : `${stale.map((c) => c.deviceName).join(', ')} belong to deleted worktrees`, '{cli} down --stale')); + + + const drift = manifestDrift(); + checks.push(check('core-drift', drift.length === 0, drift.length === 0 ? `${SHARED_ROOTS.join(', ')} match MANIFEST` : `changed: ${drift.join(', ')}`, `node src/core/manifest.ts --write, and copy ${SHARED_ROOTS.join(', ')} to clerk-android and clerk/javascript`)); + + return { verb: 'doctor', ok: checks.every((c) => c.ok), backend: { [platform]: backend.kind }, checks }; +} + +type RuntimeOutcome = LeaseOutcome & { readonly devServer: string | null; readonly instances: readonly InstanceView[] }; + +async function startRuntime(deps: Deps, lease: Lease): Promise { + if (deps.host.runtime === undefined) return null; + const runtime = await deps.host.runtime(lease, deps.progress); + const open = deps.workspace.unclosedEntries(); + for (const process of runtime.processes) { + if (open.some((e) => e.kind === 'process' && e.what === process.what && e.pid === process.pid)) continue; + deps.workspace.append({ id: newEntryId(), kind: 'process', what: process.what, pid: process.pid, startedAt: new Date(process.startedAt).toISOString(), ...(process.platform === undefined ? {} : { platform: process.platform }) }); + } + return runtime.devServer; +} + +const agentDeviceSession = (workspace: Workspace, platform: Platform) => `verify-${platform}-${workspace.worktreeId}`; + +const targetOf = (deps: Deps, outcome: RuntimeOutcome): Target => ({ + platform: outcome.lease.platform, + device: deviceOf(outcome.lease), + session: agentDeviceSession(deps.workspace, outcome.lease.platform), + build: { path: null, devServer: outcome.devServer }, +}); + +export async function up(deps: Deps, command: Extract): Promise { + const platform = platformOf(deps.host, command.platform); + chooseBackend(deps, platform, command); + return deps.workspace.withAcquireLock(platform, async (lock) => { + const [leased, instances] = await leaseWithInstances(deps, { willChange: false }, () => + ensureLease(lock, command.backend, deps.workspace, deps.host, { waitSeconds: command.waitSeconds, progress: deps.progress, instances: deps.instances, retryWith: '{cli} up --wait ' }), + ); + const outcome = { ...leased, devServer: await startRuntime(deps, leased.lease), instances }; + return { verb: 'up', leases: [outcome.view], builds: [outcome.build], instances: outcome.instances }; + }, (owner) => deps.progress(`wait another {cli} in this worktree (pid ${owner.pid}) is leasing the device; waiting for it, with no time limit`)); +} + +async function gitFacts(runner: Runner, worktree: string): Promise<{ head: string; dirty: boolean }> { + const head = await runner('git', ['rev-parse', 'HEAD'], { cwd: worktree }); + const status = await runner('git', ['status', '--porcelain'], { cwd: worktree }); + return { head: head.stdout.trim(), dirty: status.stdout.trim().length > 0 }; +} + +type Identity = EvidenceRecord['identities'][number]; + +async function newIdentities(deps: Deps, run: RunId, known: readonly Identity[]): Promise { + const seeded = deps.workspace.entries().flatMap((e): Identity[] => (e.kind === 'user' && e.run === run ? [{ email: e.email, userId: e.userId }] : [])); + const listed = await deps.instances.clerk().usersOfRun(run).catch(() => []); + const found = new Map([...listed, ...seeded].map((identity) => [identity.userId, identity] as const)); + return [...found.values()].filter((identity) => !known.some((before) => before.userId === identity.userId)); +} + +export async function endRun( + workspace: Workspace, + run: { + readonly recording: Recording | null; + readonly recorderEntry: string | null; + readonly broker: { stop(): Promise }; + readonly scratch: ScratchPath; + }, +): Promise { + const steps: (() => unknown)[] = [ + () => run.recording?.stop(), + () => { + if (run.recorderEntry !== null) workspace.append({ id: newEntryId(), kind: 'done', ref: run.recorderEntry }); + }, + () => run.broker.stop(), + () => workspace.removeScratch(run.scratch), + ]; + const errors: unknown[] = []; + for (const step of steps) { + try { + await step(); + } catch (error) { + errors.push(error); + } + } + if (errors.length > 0) throw errors[0]; +} + +export function nextStep(run: RunId, dir: EvidencePath, results: readonly SpecResult[], selection: string): string { + const ran = assertSomethingRan(results, selection); + const failed = results.filter((r) => r.status === 'failed' || r.status === 'interrupted'); + if (failed.length > 0) return failed[0]?.failurePage ?? join(dir, 'e2e.log'); + if (ran === 'all-left-out') return 'nothing ran: every selected spec was left out by platform, so the run proves nothing to post'; + return `{cli} attach ${run} --pr `; +} + +export async function leaseForRun(deps: Deps, platform: Platform, command: Extract, instances: { readonly willChange: boolean }, drive: (outcome: RuntimeOutcome) => Promise): Promise { + const key = await computeBuildKey(deps.host, platform, chooseBackend(deps, platform, command).backend.kind, deps.workspace.worktree); + const retryWith = `{cli} run ${'all' in command.selection ? '--all' : command.selection.selectors.join(' ')} --wait `; + const deviceWait = { + seconds: command.waitSeconds, + busyFix: `let the other run in this worktree finish, or rerun with a wait: ${retryWith}`, + onWait: (owner: ProcessRef) => deps.progress(`wait another {cli} run in this worktree (pid ${owner.pid}) is driving the device; waiting up to ${command.waitSeconds}s`), + }; + return deps.workspace.withAcquireThenDevice( + platform, + deviceWait, + async (lock) => { + const [leased, up] = await leaseWithInstances(deps, instances, () => + ensureLease(lock, command.backend, deps.workspace, deps.host, { waitSeconds: command.waitSeconds, progress: deps.progress, instances: deps.instances, retryWith }), + ); + const outcome: RuntimeOutcome = { ...leased, devServer: await startRuntime(deps, leased.lease), instances: up }; + deps.progress(leaseLine(outcome.view)); + return outcome; + }, + drive, + (owner) => deps.progress(`wait another {cli} in this worktree (pid ${owner.pid}) is building ${key} or leasing the device; waiting for it, with no time limit`), + ); +} + +interface GroupRun { + readonly run: RunId; + readonly dir: EvidencePath; + readonly log: EvidencePath; + readonly target: Target; + readonly broker: Pick; + readonly driver: DriverWatch; + readonly command: Extract; + readonly agent: Agent | null; +} + +interface GroupOutcome { + readonly record: EvidenceRecord['settings'][number]; + readonly results: readonly SpecResult[]; + readonly screenshots: EvidenceRecord['screenshots']; + readonly identities: readonly Identity[]; + readonly failure: VerifyFailure | null; + readonly stopsTheRun: boolean; +} + +const notRun = (spec: SpecRef, platform: Platform, why: string): SpecResult => ({ + spec, + title: 'not run', + platform, + status: 'failed', + seconds: 0, + attempts: 0, + error: redact(why), + skipReason: null, + skippedBy: null, + tags: [], + failurePage: null, + failureScreen: null, + failureScreenshot: null, +}); + +class SpecEdited extends VerifyFailure {} + +const asFailure = (error: unknown): VerifyFailure => (error instanceof VerifyFailure ? error : new VerifyFailure('NOT_READY', (error as Error).message ?? String(error), 'run `{cli} doctor`, then rerun')); + +async function runGroup(deps: Deps, the: GroupRun, group: SettingsGroup, index: number, known: readonly Identity[], stopped: VerifyFailure | null): Promise { + const { workspace } = deps; + const { settings } = group; + const specs = group.specs.map((spec): SpecRef => ({ kind: spec.kind, path: spec.path, feature: spec.feature })); + const outputDir = e2eOutputDir(the.dir, index); + const reportFile = join(outputDir, 'report.json') as EvidencePath; + let applied: AppliedInstance | null = null; + let invoked: { readonly exitCode: number } | null = null; + let held = false; + let unread: string | null = null; + let identities: readonly Identity[] = []; + let failure: VerifyFailure | null = null; + let stopsTheRun = false; + let lostItsSettings = false; + const refuseEdited = (): void => { + const edited = group.specs.find((spec) => sourceHash(readSpecText(workspace.packageDir, spec.path)) !== spec.sourceHash); + if (edited !== undefined) throw new SpecEdited('USAGE', `${edited.path} or its settings file changed while the run was in progress`, 'rerun; a run plans its groups from the spec files and their settings files as they are when it starts'); + }; + try { + if (stopped !== null) throw new VerifyFailure(stopped.code, `an earlier group of this run failed, so this one did not run: ${stopped.message}`, stopped.fix); + refuseEdited(); + try { + applied = await deps.instances.apply(group, deps.progress); + } catch (error) { + stopsTheRun = !(error instanceof SettingsRefused); + throw error; + } + try { + refuseEdited(); + if (index > 0) appendFileSync(the.log, `settings ${settings.label}: ${count(specs.length, 'spec file')}\n`); + const inputs: Inputs = { + target: the.target, + clerk: { publishableKey: deps.instances.keys().pk, access: { kind: 'stand-in', url: the.broker.url, tokenFile: the.broker.tokenFile } }, + run: the.run, + }; + the.driver.startClean(deps.progress); + invoked = await invokeE2E(planE2E(inputs, specs, the.command, workspace.packageDir, outputDir), the.log, workspace.packageDir, deps.progress, the.agent); + identities = await newIdentities(deps, the.run, known); + try { + held = await applied.stillApplied(); + } catch (error) { + unread = (error as Error).message ?? String(error); + } + refuseEdited(); + } finally { + await applied.release(); + } + } catch (error) { + failure = asFailure(error); + stopsTheRun ||= stopped === null && applied !== null && !(error instanceof SpecEdited); + } + + let results: readonly SpecResult[] = []; + let screenshots: EvidenceRecord['screenshots'] = []; + if (invoked !== null && failure === null) { + try { + const report: unknown = existsSync(reportFile) ? JSON.parse(readFileSync(reportFile, 'utf8')) : null; + if (report === null) { + const daemon = the.driver.daemonThatDidNotStart(); + throw new VerifyFailure('E2E_CRASHED', `e2e exited ${invoked.exitCode} before writing a report for ${settings.label}${daemon === null ? '' : `; ${daemon}`}`, `read ${the.log}${daemon === null ? '' : `, and driver/summary.txt beside it`}`); + } + results = parseE2EReport(report, specs, outputDir); + screenshots = collectScreenshots(report, the.dir, outputDir); + } catch (error) { + failure = error instanceof VerifyFailure ? error : new VerifyFailure('E2E_CRASHED', `e2e's report could not be read: ${(error as Error).message}`, `read ${reportFile}`); + results = []; + screenshots = []; + } + if (failure === null && !held) { + lostItsSettings = true; + failure = + unread === null + ? new VerifyFailure('INSTANCE_MISCONFIGURED', `the instance no longer showed ${settings.label} when its specs ended, so what they saw is unknown`, 'rerun; if another command in this worktree changed the instance, let one finish before the other starts') + : new VerifyFailure('NOT_READY', `the instance's environment could not be read after the specs on ${settings.label} ended, so what they saw is unknown: ${unread}`, 'rerun; a cloud environment needs *.clerk.accounts.dev in its allowed domains'); + } + } + const unreported = invoked === null || invoked.exitCode === 0 ? 'e2e reported no result for this file: it registers no test' : `e2e exited ${invoked.exitCode} and reported no result for this file: it failed to load or registers no test; e2e.log in the run directory says which`; + const why = failure?.message ?? unreported; + const missing = specs.filter((spec) => !results.some((result) => result.spec.path === spec.path)); + const reported = lostItsSettings ? specs.filter((spec) => !missing.includes(spec)) : []; + return { + record: { + label: settings.label, + askedBy: settings.askedBy, + specs: specs.map((spec) => spec.path), + application: applied?.instance.id ?? null, + changed: applied !== null && applied.changed, + held, + e2eReport: invoked === null ? null : reportFile, + }, + results: [...results, ...[...missing, ...reported].map((spec) => notRun(spec, the.target.platform, why))], + screenshots, + identities, + failure, + stopsTheRun, + }; +} + +export async function runVerb(deps: Deps, command: Extract): Promise { + const { host, workspace } = deps; + const platform = platformOf(host, command.platform); + const specs = resolveSpecs(workspace.packageDir, command.selection); + protectGitHubTokens(deps.env); + const agent = deps.agent?.() ?? null; + const recorded = deps.instances.recordedKey(); + const stray = straySettingsFile(workspace.packageDir); + if (stray !== null) throw stray; + const sources = specs.map((spec) => ({ spec, ...readSpecText(workspace.packageDir, spec.path) })); + const groups = planGroups(sources, recorded ?? STANDARD.key); + if (groups.length > 1) deps.progress(`settings ${groups.length} groups in this run: ${groups.map((group, index) => `${group.settings.label} (${index === 0 ? count(group.specs.length, 'spec file') : group.specs.length})`).join(', ')}`); + return leaseForRun(deps, platform, command, { willChange: groups.some((group) => group.settings.key !== recorded) }, async (outcome) => { + try { + const { lease, backend } = outcome; + const { run, dir, scratch } = workspace.newRun(); + runPassword(run); + const driver = (deps.watchDriver ?? watchDriver)(workspace.agentDeviceDir, dir); + const startedAt = new Date(); + deps.progress(`run ${run} ${platform} ${outcome.view.backend} ${outcome.view.device} build ${outcome.app.key}`); + for (const { spec, source, declaration } of sources) { + mkdirSync(dirname(join(dir, spec.path)), { recursive: true }); + writeFileSync(join(dir, spec.path), source); + if (declaration !== null) writeFileSync(join(dir, settingsFileOf(spec.path)), declaration); + } + + const broker = await startBroker(run, workspace, scratch, { keys: () => deps.instances.keys(), fetch: deps.fetch ?? fetch }); + + let recording: Recording | null = null; + let recorderEntry: string | null = null; + const outcomes: GroupOutcome[] = []; + let endFailure: unknown = null; + try { + if (command.video) { + recording = await backend.startRecording(lease, dir); + if (recording.process !== null) { + recorderEntry = newEntryId(); + workspace.append({ id: recorderEntry, kind: 'process', what: 'recorder', pid: recording.process.pid, startedAt: new Date(recording.process.startedAt).toISOString(), platform }); + } + } + const the: GroupRun = { run, dir, log: join(dir, 'e2e.log') as EvidencePath, target: targetOf(deps, outcome), broker, driver, command, agent }; + for (const [index, group] of groups.entries()) { + const stopped = outcomes.find((done) => done.stopsTheRun)?.failure ?? null; + outcomes.push(await runGroup(deps, the, group, index, outcomes.flatMap((done) => done.identities), stopped)); + } + } finally { + await endRun(workspace, { recording, recorderEntry, broker, scratch }) + .catch((error: unknown) => void (endFailure = error)) + .finally(() => deps.instances.stopDriving()); + } + + driver.collect(); + const appLog = join(dir, 'app.log') as EvidencePath; + const logs = await backend.logs(lease, startedAt, host.logPredicates?.[platform]).catch((error: unknown) => `the device logs could not be read: ${(error as Error).message ?? String(error)}`); + writeFileSync(appLog, redact(logs)); + + const results = outcomes.flatMap((done) => done.results); + const screenshots = new Map(outcomes.flatMap((done) => done.screenshots.map((shot) => [shot.label, shot] as const))); + const git = await gitFacts(deps.runner, workspace.worktree); + const record = sealEvidence(dir, { + run, + startedAt: startedAt.toISOString(), + finishedAt: new Date().toISOString(), + repo: host.repo, + gitHead: git.head, + dirty: git.dirty, + platform, + backend: lease.backend, + device: outcome.view.device, + build: outcome.app.key, + results, + videos: existsSync(join(dir, 'video.mp4')) ? [join(dir, 'video.mp4') as EvidencePath] : [], + screenshots: [...screenshots.values()], + appLog, + e2eReport: join(e2eOutputDir(dir, 0), 'report.json') as EvidencePath, + identities: outcomes.flatMap((done) => done.identities), + settings: outcomes.map((done) => done.record), + }); + const failed = outcomes.flatMap((done) => (done.failure === null ? [] : [{ label: done.record.label, failure: done.failure }])); + if (command.githubReport) { + for (const line of await reportToGitHub({ record, failures: failed, packageDir: workspace.packageDir })) deps.progress(`github ${line}`); + } + if (failed[0] !== undefined) { + deps.progress(`evidence ${dir} sealed; ${count(failed.length, 'group')} of ${groups.length} did not run in full, and run.json has a failed result for each of their spec files`); + throw failed[0].failure; + } + if (endFailure !== null) { + deps.progress(`evidence ${dir} sealed; the specs ran, and the run still fails because it did not end cleanly`); + throw endFailure; + } + const next = nextStep(run, dir, results, 'all' in command.selection ? '--all' : command.selection.selectors.join(' ')); + return { verb: 'run', dir, record, next }; + } finally { + ledgerAgentDeviceDaemon(workspace); + } + }); +} + +interface SnapshotNode { + readonly kind?: string; + readonly type?: string; + readonly label?: string; + readonly value?: string; + readonly identifier?: string; + readonly depth?: number; +} + +function heldLease(deps: Deps, platform: Platform): { lease: Lease; backend: DeviceBackend } { + const lease = deps.workspace.readLease(platform); + if (lease === null) throw new VerifyFailure('NOT_READY', `this worktree holds no ${platform} device`, '{cli} up'); + return { lease, backend: backendFor(deps.host, platform, lease.backend) }; +} + +function screenNodes(snapshot: readonly SnapshotNode[]): readonly ScreenNode[] { + const counts = new Map(); + for (const node of snapshot) if (node.identifier) counts.set(node.identifier, (counts.get(node.identifier) ?? 0) + 1); + return snapshot.map((node) => { + const testId = node.identifier || null; + const text = node.value || null; + return { + role: (node.kind ?? node.type ?? 'node').toLowerCase(), + name: node.label || null, + testId, + text, + depth: node.depth ?? 0, + locator: testId !== null && counts.get(testId) === 1 ? `screen.getByTestId('${testId}')` : null, + }; + }); +} + +async function screen(deps: Deps, command: Extract): Promise { + const platform = platformOf(deps.host, command.platform); + const { lease, backend } = heldLease(deps, platform); + if ((await backend.check(lease)) === 'lost') throw new VerifyFailure('LEASE_LOST', `${backend.describe(lease)} is gone`, '{cli} up'); + const env = withoutClerkKeys(deps.env); + const target = agentDeviceOn(deviceOf(lease), platform); + const agentDevice = (args: readonly string[]) => + deps.runner(join(deps.workspace.packageDir, 'node_modules', '.bin', 'agent-device'), args, { env: { ...env, AGENT_DEVICE_STATE_DIR: deps.workspace.agentDeviceDir, ...target.env } }); + const screenWait = { seconds: 10, busyFix: 'let the run in this worktree finish, then rerun {cli} screen' }; + return deps.workspace.withDevice(platform, screenWait, async () => { + const selector = target.selector; + const session = ['--session', `${agentDeviceSession(deps.workspace, platform)}-screen`]; + const attachedWithoutRelaunch = await agentDevice(['open', deps.host.appId(platform), '--json', ...selector, ...session]); + if (attachedWithoutRelaunch.code !== 0) throw new VerifyFailure('NOT_READY', `agent-device open failed: ${redact(attachedWithoutRelaunch.stdout.trim() || attachedWithoutRelaunch.stderr.trim())}`, '{cli} up, then retry'); + const snap = await agentDevice(['snapshot', '--json', ...selector, ...session]); + if (snap.code !== 0) throw new VerifyFailure('NOT_READY', `agent-device snapshot failed: ${redact(snap.stderr.trim() || snap.stdout.trim())}`, 'run a spec first so the app is open, then retry'); + const parsed = JSON.parse(snap.stdout) as { data?: { nodes?: SnapshotNode[] } }; + const nodes = screenNodes(parsed.data?.nodes ?? []); + let png: ScratchPath | null = null; + if (command.png) { + const dir = join(deps.workspace.root, 'scratch', 'screens'); + mkdirSync(dir, { recursive: true }); + png = join(dir, `${new Date().toISOString().replace(/[:.]/g, '-')}.png`) as ScratchPath; + const shot = await agentDevice(['screenshot', png, ...selector, ...session]); + if (shot.code !== 0) throw new VerifyFailure('NOT_READY', `agent-device screenshot failed: ${redact(shot.stderr.trim())}`, 'retry, or check `{cli} doctor`'); + } + await agentDevice(['close', ...selector, ...session]); + ledgerAgentDeviceDaemon(deps.workspace); + return { verb: 'screen', platform, device: backend.describe(lease), nodes, png }; + }); +} + +export async function attach(deps: Deps, command: Extract): Promise { + const run = parseRunId(command.run); + const dir = deps.workspace.runDir(run); + if (!existsSync(dir)) throw new VerifyFailure('USAGE', `no run ${run} in ${deps.workspace.root}/runs`, 'pass a run id that `{cli} run` printed'); + const record = readRecord(dir); + const publishable = assertPublishable(record, loggedUserIds(dir)); + return postToPullRequest(publishable, dir, deps.host, command.pr, command.screenshots, deps.runner); +} + +interface DownPlan { + readonly leases: readonly { readonly lease: Lease; readonly backend: DeviceBackend; readonly view: LeaseView; readonly origin: 'lease-file' | 'stale-claim' }[]; + readonly stillLeased: readonly Platform[]; + readonly processes: readonly ProcessEntry[]; + readonly staleIntents: readonly string[]; +} + +const leaseIdentity = (lease: Lease): string => `local:${lease.claimNonce}`; + +async function planDown(deps: Deps, command: Extract): Promise { + const { host, workspace } = deps; + const platforms = command.platform === undefined ? host.platforms : [platformOf(host, command.platform)]; + const leases: DownPlan['leases'][number][] = []; + for (const platform of platforms) { + const lease = workspace.readLease(platform); + if (lease !== null) { + const backend = backendFor(host, platform, lease.backend); + leases.push({ lease, backend, view: leaseView(backend, lease, false), origin: 'lease-file' }); + } + if (command.stale) { + for (const backend of host.backends.filter((b) => b.platform === platform)) { + for (const orphan of await backend.reapable(workspace.worktree)) { + if (leases.some((l) => leaseIdentity(l.lease) === leaseIdentity(orphan))) continue; + leases.push({ lease: orphan, backend, view: leaseView(backend, orphan, false), origin: 'stale-claim' }); + } + } + } + } + const stillLeased = host.platforms.filter((platform) => !platforms.includes(platform) && workspace.readLease(platform) !== null); + return { + leases, + stillLeased, + processes: processesIn(workspace, { platforms, sharedByEveryLease: stillLeased.length === 0 }), + staleIntents: command.stale ? workspace.unclosedEntries().filter((e) => e.kind === 'lease-intent').map((e) => e.id) : [], + }; +} + +export function down(deps: Deps, command: Extract): Promise { + if (command.dryRun) return downUnlocked(deps, command); + const platforms = command.platform === undefined ? deps.host.platforms : [platformOf(deps.host, command.platform)]; + const locked = platforms.reduce<() => Promise>( + (inner, platform) => () => + deps.workspace.withAcquireLock( + platform, + () => + deps.workspace.withDevice( + platform, + { + seconds: Number.POSITIVE_INFINITY, + busyFix: '', + onWait: (owner) => deps.progress(`wait another {cli} run in this worktree (pid ${owner.pid}) is driving the device; down waits for it, with no time limit`), + }, + inner, + ), + (owner) => deps.progress(`wait another {cli} in this worktree (pid ${owner.pid}) is building or leasing the device; down waits for it, with no time limit`), + ), + () => downUnlocked(deps, command), + ); + return locked(); +} + +function runningDaemon(workspace: Workspace): readonly string[] { + const daemon = readDaemonInfo(workspace.agentDeviceDir); + return daemon !== null && isRunning(daemon) ? [`agent-device ${daemon.pid}`] : []; +} + +async function downUnlocked(deps: Deps, command: Extract): Promise { + const { workspace } = deps; + if (!command.dryRun) ledgerAgentDeviceDaemon(workspace); + const plan = await planDown(deps, command); + const sharedStays = plan.stillLeased.length > 0; + if (command.dryRun) { + return { + verb: 'down', + dryRun: true, + wouldRelease: plan.leases.map((l) => l.view), + wouldDelete: sharedStays ? [] : openApplications(workspace).map((entry) => ({ kind: 'application', name: entry.name })), + wouldStop: [...new Set([...plan.processes.filter((p) => isRunning({ pid: p.pid, startedAt: Date.parse(p.startedAt) })).map((p) => `${p.what} ${p.pid}`), ...(sharedStays ? [] : runningDaemon(workspace))])], + keptRuns: workspace.runs(), + }; + } + const stoppedProcesses = stopProcesses(workspace, plan.processes); + const unreleased: unknown[] = []; + for (const { lease, backend, origin } of plan.leases) { + await (origin === 'lease-file' ? releaseLease(workspace, backend, lease) : backend.release(lease)).catch((error: unknown) => unreleased.push(error)); + } + if (sharedStays && openApplications(workspace).length > 0) deps.progress(`kept this worktree's throwaway instances, which its ${plan.stillLeased.join(' and ')} lease still uses`); + const deleted = await deps.instances.finish(workspace, { keepApplications: sharedStays }, deps.progress).catch((error: unknown) => { + unreleased.push(error); + return []; + }); + for (const ref of plan.staleIntents) workspace.append({ id: newEntryId(), kind: 'done', ref }); + if (unreleased.length > 0) throw unreleased[0]; + return { + verb: 'down', + dryRun: false, + released: plan.leases.map((l) => l.view), + deletedApplications: deleted, + stoppedProcesses, + keptRuns: workspace.runs(), + }; +} + +export const verbs = { doctor, up, run: runVerb, screen, attach, down } as const; diff --git a/integration/expo-native/src/core/workspace.ts b/integration/expo-native/src/core/workspace.ts new file mode 100644 index 00000000000..7dc69c29419 --- /dev/null +++ b/integration/expo-native/src/core/workspace.ts @@ -0,0 +1,232 @@ +import { createHash, randomUUID } from 'node:crypto'; +import { appendFileSync, existsSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { join, relative, resolve } from 'node:path'; +import { isRunId, newRunId } from '../../specs/support/inputs.ts'; +import { currentProcess, isRunning, sleep, type ProcessRef } from './exec.ts'; +import { compareAndSwapSlot, readSlot } from './slot.ts'; +import { + VerifyFailure, + type AcquireLock, + type DeviceWait, + type EvidencePath, + type Lease, + type LedgerEntry, + type Platform, + type RunId, + type ScratchPath, +} from './types.ts'; + +export interface WorkspaceOptions { + readonly packageDir: string; + readonly worktree: string; + readonly home?: string; +} + +export interface Workspace { + readonly root: string; + readonly packageDir: string; + readonly worktree: string; + readonly worktreeId: string; + readonly home: string; + readonly ledgerFile: string; + readonly claimsDir: string; + readonly agentDeviceDir: string; + newRun(): { readonly run: RunId; readonly dir: EvidencePath; readonly scratch: ScratchPath }; + runDir(run: RunId): EvidencePath; + runs(): readonly RunId[]; + buildsDir(): ScratchPath; + leaseFile(platform: Platform): string; + readLease(platform: Platform): Lease | null; + writeLease(lease: Lease): void; + clearLease(platform: Platform): void; + append(entry: LedgerEntry): void; + entries(): readonly LedgerEntry[]; + unclosedEntries(): readonly LedgerEntry[]; + withAcquireLock(platform: Platform, fn: (lock: AcquireLock) => Promise, onWait?: (owner: ProcessRef) => void): Promise; + withDevice(platform: Platform, wait: DeviceWait, fn: () => Promise): Promise; + withAcquireThenDevice( + platform: Platform, + deviceWait: DeviceWait, + prepare: (lock: AcquireLock) => Promise, + drive: (prepared: A) => Promise, + onAcquireWait?: (owner: ProcessRef) => void, + ): Promise; + removeScratch(path: ScratchPath): void; +} + +export const newEntryId = (): string => randomUUID(); + +export const WORKSPACE_DIR = '.verify'; + +export const agentDeviceStateDir = (workspaceRoot: string): string => join(workspaceRoot, 'agent-device'); + +function worktreeIdOf(worktree: string): string { + return createHash('sha256').update(resolve(worktree)).digest('hex').slice(0, 12); +} + +export function parseRunId(value: string): RunId { + if (!isRunId(value)) throw new VerifyFailure('USAGE', `${value} is not a run id`, 'pass an id like r20261002-141210-7c1e from `{cli} run`'); + return value; +} + +function isPlatform(value: unknown): value is Platform { + return value === 'ios' || value === 'android'; +} + +function parseLease(text: string, file: string): Lease { + const raw: unknown = JSON.parse(text); + const bad = () => new VerifyFailure('LEASE_LOST', `${file} is not a lease`, '{cli} down, then {cli} up'); + if (typeof raw !== 'object' || raw === null) throw bad(); + const r = raw as Record; + if (!isPlatform(r.platform) || typeof r.acquiredAt !== 'string') throw bad(); + const installedBuild = typeof r.installedBuild === 'string' ? r.installedBuild : null; + if (r.backend === 'local') { + if (typeof r.slot !== 'number' || typeof r.deviceName !== 'string' || typeof r.deviceId !== 'string' || typeof r.claimNonce !== 'string') throw bad(); + if (r.deviceName !== `verify-${r.platform}-${r.slot}`) throw bad(); + return { ...(r as object), installedBuild } as Lease; + } + throw bad(); +} + +function writePrivate(file: string, text: string): void { + writeFileSync(file, text, { mode: 0o600 }); +} + +function lockOwner(value: string | null): ProcessRef | null { + if (value === null) return null; + try { + const parsed: unknown = JSON.parse(value); + if (typeof parsed !== 'object' || parsed === null) return null; + const { pid, startedAt } = parsed as Partial; + return typeof pid === 'number' && typeof startedAt === 'number' ? { pid, startedAt } : null; + } catch { + return null; + } +} + +export async function takeSlotLock(dir: string, timeoutMs: number, onTimeout: () => VerifyFailure, onWait?: (owner: ProcessRef) => void): Promise<() => void> { + const deadline = Date.now() + timeoutMs; + const me = currentProcess(); + let announced = false; + for (;;) { + const state = readSlot(dir); + const owner = lockOwner(state.value); + const running = owner !== null && isRunning(owner); + if (!running && compareAndSwapSlot(dir, state.gen, JSON.stringify(me))) { + const held = state.gen + 1; + return () => void compareAndSwapSlot(dir, held, null); + } + if (owner !== null && running) { + if (Date.now() >= deadline) throw onTimeout(); + if (!announced && onWait !== undefined) onWait(owner); + announced = true; + await sleep(250); + } + } +} + +async function withSlotLock(dir: string, timeoutMs: number, onTimeout: () => VerifyFailure, fn: () => Promise, onWait?: (owner: ProcessRef) => void): Promise { + const release = await takeSlotLock(dir, timeoutMs, onTimeout, onWait); + try { + return await fn(); + } finally { + release(); + } +} + +const deviceBusy = (platform: Platform, fix: string) => + new VerifyFailure('DEVICE_BUSY', `another {cli} process in this worktree is driving the ${platform} device`, fix); + +export function openWorkspace(options: WorkspaceOptions): Workspace { + const root = join(options.packageDir, WORKSPACE_DIR); + const home = options.home ?? join(homedir(), '.verify'); + const worktreeId = worktreeIdOf(options.worktree); + const ledgerFile = join(home, 'ledgers', `${worktreeId}.jsonl`); + const claimsDir = join(home, 'claims'); + const dir = (...parts: string[]) => { + const path = join(root, ...parts); + mkdirSync(path, { recursive: true }); + return path; + }; + const readEntries = (): LedgerEntry[] => { + if (!existsSync(ledgerFile)) return []; + return readFileSync(ledgerFile, 'utf8') + .split('\n') + .filter((line) => line.trim().length > 0) + .map((line) => JSON.parse(line) as LedgerEntry); + }; + + const acquireDir = (platform: Platform) => join(dir('locks'), `acquire-${platform}`); + const unreachable = () => new VerifyFailure('DEVICE_BUSY', 'unreachable', ''); + + return { + root, + packageDir: options.packageDir, + worktree: options.worktree, + worktreeId, + home, + ledgerFile, + claimsDir, + agentDeviceDir: agentDeviceStateDir(root), + newRun() { + const run = newRunId(); + return { run, dir: dir('runs', run) as EvidencePath, scratch: dir('scratch', run) as ScratchPath }; + }, + runDir: (run) => join(root, 'runs', run) as EvidencePath, + runs: () => (existsSync(join(root, 'runs')) ? readdirSync(join(root, 'runs')).filter(isRunId).sort() : []), + buildsDir: () => join(root, 'builds') as ScratchPath, + leaseFile: (platform) => join(root, 'leases', `${platform}.json`), + readLease(platform) { + const file = join(root, 'leases', `${platform}.json`); + return existsSync(file) ? parseLease(readFileSync(file, 'utf8'), file) : null; + }, + writeLease(lease) { + writePrivate(join(dir('leases'), `${lease.platform}.json`), `${JSON.stringify(lease, null, 2)}\n`); + }, + clearLease(platform) { + rmSync(join(root, 'leases', `${platform}.json`), { force: true }); + }, + append(entry) { + mkdirSync(join(home, 'ledgers'), { recursive: true }); + const owner = join(home, 'ledgers', `${worktreeId}.owner`); + if (!existsSync(owner)) writePrivate(owner, `${resolve(options.worktree)}\n${resolve(options.packageDir)}\n`); + appendFileSync(ledgerFile, `${JSON.stringify(entry)}\n`, { mode: 0o600, flag: 'a' }); + }, + entries: readEntries, + unclosedEntries() { + const entries = readEntries(); + const closed = new Set(entries.flatMap((e) => (e.kind === 'done' ? [e.ref] : []))); + return entries.filter((e) => e.kind !== 'done' && !closed.has(e.id)); + }, + withAcquireLock(platform, fn, onWait) { + return withSlotLock(acquireDir(platform), Number.POSITIVE_INFINITY, unreachable, () => fn({ platform } as AcquireLock), onWait); + }, + withDevice(platform, wait, fn) { + return withSlotLock(join(dir('locks'), `device-${platform}`), wait.seconds * 1000, () => deviceBusy(platform, wait.busyFix), fn, wait.onWait); + }, + async withAcquireThenDevice(platform, deviceWait, prepare, drive, onAcquireWait) { + const releaseAcquire = await takeSlotLock(acquireDir(platform), Number.POSITIVE_INFINITY, unreachable, onAcquireWait); + let prepared; + let releaseDevice; + try { + prepared = await prepare({ platform } as AcquireLock); + releaseDevice = await takeSlotLock(join(dir('locks'), `device-${platform}`), deviceWait.seconds * 1000, () => deviceBusy(platform, deviceWait.busyFix), deviceWait.onWait); + } finally { + releaseAcquire(); + } + try { + return await drive(prepared); + } finally { + releaseDevice(); + } + }, + removeScratch(path) { + const rel = relative(root, path); + if (!(rel.startsWith('scratch') || rel.startsWith('builds')) || rel.includes('..')) { + throw new VerifyFailure('EVIDENCE_UNSAFE', `${path} is not scratch`, 'only .verify/scratch and .verify/builds are deletable'); + } + rmSync(path, { recursive: true, force: true }); + }, + }; +} diff --git a/integration/expo-native/testing/claim-taker.ts b/integration/expo-native/testing/claim-taker.ts new file mode 100644 index 00000000000..b61506805b8 --- /dev/null +++ b/integration/expo-native/testing/claim-taker.ts @@ -0,0 +1,5 @@ +import { takeSlot } from '../src/core/claims.ts'; + +const [dir, worktree, startAt, from] = process.argv.slice(2) as [string, string, string, string]; +while (Date.now() < Number(startAt)) await new Promise((resolve) => setTimeout(resolve, 1)); +process.stdout.write(takeSlot(dir, 'ios', 1, Number(from), worktree) === null ? 'lost' : 'won'); diff --git a/integration/expo-native/testing/fake-clerk.ts b/integration/expo-native/testing/fake-clerk.ts new file mode 100644 index 00000000000..c481af72e0f --- /dev/null +++ b/integration/expo-native/testing/fake-clerk.ts @@ -0,0 +1,247 @@ +import { configLeaves, standardFile } from '../src/core/instances/settings.ts'; +import type { Json } from '../src/core/types.ts'; + +export const WORKSPACE = 'org_3KHungJxbvIscuSvy8oos5MHAli'; +export const PLATFORM_KEY = 'ak_unitTestPlatformKey0123456789'; +export const SECRET_KEY_SCOPE = 'application_secret_keys:read'; +export const PLATFORM_SCOPES: readonly string[] = ['applications:read', 'applications:manage', 'applications:delete', SECRET_KEY_SCOPE]; + +interface FakeApplication { + readonly id: string; + readonly name: string; + readonly instanceId: string; + readonly pk: string; + readonly sk: string; + config: Record; + environment: Record; + users: number; + created: { readonly id: string; readonly email_addresses: readonly { readonly email_address: string }[] }[]; + deleted: boolean; +} + +type JsonObject = { readonly [key: string]: Json }; +const isObject = (value: Json | undefined): value is JsonObject => typeof value === 'object' && value !== null && !Array.isArray(value); + +function merged(base: JsonObject, over: JsonObject): Record { + const out: Record = { ...base }; + for (const [key, value] of Object.entries(over)) { + const under = out[key]; + out[key] = isObject(value) && isObject(under) ? merged(under, value) : value; + } + return out; +} + +const SHOWN: readonly { readonly config: string; readonly leaf: string; readonly as?: (value: Json) => Json }[] = [ + { config: 'organization_settings.force_organization_selection', leaf: 'organization_settings.force_organization_selection' }, + { config: 'auth_multi_factor.required_for_sign_up', leaf: 'user_settings.sign_up.mfa.required' }, + { config: 'session_settings.multi_session_enabled', leaf: 'auth_config.single_session_mode', as: (value) => value !== true }, + { config: 'auth_password.min_length', leaf: 'user_settings.password_settings.min_length' }, +]; + +export interface FakeRefusal { + readonly path: string; + readonly value?: Json; + readonly status: number; + readonly code: string; + readonly param?: string | readonly string[]; + readonly message: string; +} + +export interface FakeClerkOptions { + readonly workspace?: string; + readonly attachesKey?: boolean; + readonly replacesAuthorizationOnCom?: boolean; + readonly now?: () => number; +} + +function unflatten(leaves: Readonly>): Json { + const root: Record = {}; + for (const [path, value] of Object.entries(leaves)) { + const parts = path.split('.').flatMap((part) => { + const match = /^([^[]+)((?:\[\d+\])*)$/.exec(part); + if (match === null) return [part]; + return [match[1]!, ...[...match[2]!.matchAll(/\[(\d+)\]/g)].map((index) => Number(index[1]))]; + }); + let node = root as Record; + parts.forEach((part, index) => { + if (index === parts.length - 1) node[part] = value; + else node = (node[part] ??= typeof parts[index + 1] === 'number' ? [] : {}) as Record; + }); + } + return root as Json; +} + +const same = (a: unknown, b: unknown): boolean => JSON.stringify(a) === JSON.stringify(b); + +const nested = (path: string, value: Json): JsonObject => path.split('.').reduceRight((inner, key) => ({ [key]: inner }), value) as JsonObject; + +export function fakeClerk(options: FakeClerkOptions = {}) { + const now = options.now ?? Date.now; + const applications: FakeApplication[] = []; + const requests: { readonly method: string; readonly url: string; readonly authorization: string | null; readonly body: unknown }[] = []; + let counter = 0; + let rateLimited: { left: number; retryAfter: string | null } = { left: 0, retryAfter: null }; + const state = { + listShape: 'array' as 'array' | 'envelope' | 'unreadable', + pageSize: 2, + failConfigure: 0, + loseCreateAnswer: false, + refuseDelete: false, + noDate: false, + refuseInstanceKeys: false, + scopes: PLATFORM_SCOPES as readonly string[] | undefined, + ignoreConfigKey: null as string | null, + refusals: [] as FakeRefusal[], + alsoMoves: {} as Record, + dropFromAfter: null as string | null, + storesInstead: {} as Record, + }; + + const json = (status: number, body: unknown, headers: Record = {}): Response => + new Response(JSON.stringify(body), { status, headers: { ...(state.noDate ? {} : { date: new Date(now()).toUTCString() }), ...headers } }); + const error = (status: number, code: string): Response => json(status, { errors: [{ code, message: code }] }); + const live = () => applications.filter((application) => !application.deleted); + + const freshEnvironment = (): Record => ({ 'auth_config.reverification': true, ...standardFile().environment, 'auth_config.single_session_mode': true, 'user_settings.password_settings.min_length': 15 }); + + const fresh = (): Pick => ({ config: {}, environment: freshEnvironment(), users: 0, created: [], deleted: false }); + + function platform(method: string, path: string, authorization: string | null, body: unknown): Response { + if (rateLimited.left > 0) { + rateLimited.left -= 1; + return json(429, { errors: [{ code: 'too_many_requests' }] }, rateLimited.retryAfter === null ? {} : { 'retry-after': rateLimited.retryAfter }); + } + if (authorization === null) return error(401, 'authorization_header_format_invalid'); + if (authorization !== `Bearer ${PLATFORM_KEY}`) return error(401, 'could_not_authenticate_request'); + if (method === 'GET' && path === '/me') return json(200, { object: 'platform_principal', subject: options.workspace ?? WORKSPACE, actor: PLATFORM_KEY, ...(state.scopes === undefined ? {} : { scopes: state.scopes }) }); + const listed = (application: FakeApplication, withSecretKey: boolean): { readonly application_id: string; readonly name: string; readonly instances: readonly Readonly>[] } => ({ + application_id: application.id, + name: application.name, + instances: [{ environment_type: 'development', instance_id: application.instanceId, ...(withSecretKey ? { secret_key: application.sk } : {}), publishable_key: application.pk }], + }); + const read = /^\/applications\/([^/?]+)(\?include_secret_keys=true)?$/.exec(path); + if (method === 'GET' && read !== null) { + if (read[2] !== undefined && state.scopes?.includes(SECRET_KEY_SCOPE) !== true) return json(403, { errors: [{ code: 'authorization_missing_scopes', message: 'Missing scopes', meta: { scopes: [SECRET_KEY_SCOPE] } }] }); + const application = live().find((candidate) => candidate.id === read[1]); + return application === undefined ? error(404, 'resource_not_found') : json(200, listed(application, read[2] !== undefined)); + } + if (method === 'GET' && path.startsWith('/applications')) { + const all = live().map((application) => listed(application, false)); + if (state.listShape === 'array') return json(200, all); + if (state.listShape === 'unreadable') return json(200, { applications: all }); + const offset = Number(new URL(`https://x${path}`).searchParams.get('offset') ?? 0); + return json(200, { data: all.slice(offset, offset + state.pageSize), total_count: all.length }); + } + if (method === 'POST' && path === '/applications') { + counter += 1; + const slug = `fake-${counter}`; + const application: FakeApplication = { + id: `app_fake${counter}`, + name: (body as { name: string }).name, + instanceId: `ins_fake${counter}`, + pk: `pk_test_${Buffer.from(`${slug}.clerk.accounts.dev$`).toString('base64url')}`, + sk: `sk_test_fakeSecret${counter}xxxxxxxxxxxxxxxx`, + ...fresh(), + }; + applications.push(application); + if (state.loseCreateAnswer) return error(500, 'internal'); + return json(200, { application_id: application.id, name: application.name, instances: [{ environment_type: 'development', instance_id: application.instanceId, publishable_key: application.pk, secret_key: application.sk }] }); + } + const config = /^\/applications\/([^/]+)\/instances\/([^/?]+)\/config(\?dry_run=true)?$/.exec(path); + if (method === 'PATCH' && config !== null) { + if (state.failConfigure > 0) { + state.failConfigure -= 1; + return error(500, 'internal'); + } + const application = live().find((candidate) => candidate.id === config[1]); + if (application === undefined) return error(404, 'resource_not_found'); + const sent = body as JsonObject; + const leaves = configLeaves(sent); + const refusal = state.refusals.find((candidate) => candidate.path in leaves && (candidate.value === undefined || same(leaves[candidate.path], candidate.value))); + if (refusal !== undefined) { + const named = refusal.param === undefined ? [undefined] : [refusal.param].flat(); + return json(refusal.status, { errors: named.map((param) => ({ code: refusal.code, message: refusal.message, long_message: refusal.message, ...(param === undefined ? {} : { meta: { param_name: param } }) })) }); + } + let after = merged(application.config, sent); + for (const [path, value] of Object.entries(state.storesInstead)) if (path in leaves) after = merged(after, nested(path, value)); + const named = (from: JsonObject): JsonObject => Object.fromEntries(Object.keys(sent).flatMap((key) => (key in from ? [[key, from[key]!]] : []))); + const reordered = (value: Json): Json => (Array.isArray(value) ? [...value].sort().reverse() : typeof value === 'object' && value !== null ? Object.fromEntries(Object.entries(value).map(([key, child]) => [key, reordered(child)])) : value); + const answer = { config_version: '2026-10-05', dry_run: config[3] !== undefined, before: named(application.config), after: reordered(named(after)) }; + if (state.dropFromAfter !== null) { + const [top, ...rest] = state.dropFromAfter.split('.'); + const pruned = structuredClone(answer.after) as Record>; + let node: Record | undefined = pruned[top!]; + for (const part of rest.slice(0, -1)) node = node?.[part] as Record | undefined; + if (node !== undefined) delete node[rest.at(-1)!]; + answer.after = pruned; + } + if (config[3] !== undefined) return json(200, answer); + application.config = after; + const set = configLeaves(after); + for (const shown of SHOWN) { + if (!(shown.config in set) || shown.leaf === state.ignoreConfigKey) continue; + application.environment[shown.leaf] = shown.as === undefined ? set[shown.config]! : shown.as(set[shown.config]!); + } + Object.assign(application.environment, state.alsoMoves); + return json(200, answer); + } + const one = /^\/applications\/([^/?]+)$/.exec(path); + if (method === 'DELETE' && one !== null) { + if (state.refuseDelete) return error(403, 'authorization_missing_scopes'); + const application = applications.find((candidate) => candidate.id === one[1]); + if (application === undefined) return error(404, 'resource_not_found'); + application.deleted = true; + return json(200, { id: application.id, object: 'application', deleted: true }); + } + return error(404, 'resource_not_found'); + } + + const fetchImpl = (async (input: string | URL, init?: RequestInit) => { + const url = new URL(String(input)); + const method = init?.method ?? 'GET'; + const sent = new Headers(init?.headers).get('authorization'); + const body: unknown = typeof init?.body === 'string' ? JSON.parse(init.body) : undefined; + requests.push({ method, url: `${url.host}${url.pathname}${url.search}`, authorization: sent, body }); + const onCom = url.host === 'api.clerk.com'; + if (onCom && url.pathname.startsWith('/v1/platform/')) { + const authorization = options.attachesKey === true ? `Bearer ${PLATFORM_KEY}` : sent; + return platform(method, `${url.pathname.slice('/v1/platform'.length)}${url.search}`, authorization, body); + } + if (onCom) { + const authorization = options.replacesAuthorizationOnCom === true ? `Bearer ${PLATFORM_KEY}` : sent; + const application = live().find((candidate) => authorization === `Bearer ${candidate.sk}`); + if (application === undefined || state.refuseInstanceKeys) return error(401, 'clerk_key_invalid'); + if (method === 'GET' && url.pathname === '/v1/users/count') return json(200, { object: 'total_count', total_count: application.users }); + if (method === 'POST' && url.pathname === '/v1/users') { + if (application.users >= 100) return error(403, 'user_quota_exceeded'); + application.users += 1; + const user = { id: `user_${application.id}_${application.users}`, email_addresses: ((body as { email_address?: string[] } | undefined)?.email_address ?? []).map((email_address) => ({ email_address })) }; + application.created.push(user); + return json(200, user); + } + if (method === 'POST' && url.pathname === '/v1/sign_in_tokens') return json(200, { token: `ticket_${application.id}_${(counter += 1)}` }); + const query = url.searchParams.get('email_address_query'); + if (method === 'GET' && url.pathname === '/v1/users' && query !== null) return json(200, application.created.filter((user) => user.email_addresses.some((address) => address.email_address.includes(query)))); + return json(200, []); + } + const application = applications.find((candidate) => `${Buffer.from(candidate.pk.slice('pk_test_'.length), 'base64url').toString().replace(/\$$/, '')}` === url.host); + if (application === undefined || application.deleted) return error(404, 'resource_not_found'); + return json(200, unflatten(application.environment)); + }) as typeof fetch; + + return { + fetch: fetchImpl, + requests, + state, + applications, + live, + platformRequests: () => requests.filter((request) => request.url.startsWith('api.clerk.com/v1/platform/')), + rateLimit: (times: number, retryAfter: string | null = null) => (rateLimited = { left: times, retryAfter }), + plant(name: string, secretKey?: string): FakeApplication { + counter += 1; + const application: FakeApplication = { id: `app_planted${counter}`, name, instanceId: `ins_planted${counter}`, pk: `pk_test_${Buffer.from(`planted-${counter}.clerk.accounts.dev$`).toString('base64url')}`, sk: secretKey ?? `sk_test_planted${counter}xxxxxxxxxxxxxxxxxxx`, ...fresh() }; + applications.push(application); + return application; + }, + }; +} diff --git a/integration/expo-native/testing/fake-instances.ts b/integration/expo-native/testing/fake-instances.ts new file mode 100644 index 00000000000..a18ef8029e4 --- /dev/null +++ b/integration/expo-native/testing/fake-instances.ts @@ -0,0 +1,26 @@ +import type { Instances } from '../src/core/instances/instances.ts'; +import type { ApplicationView, InstanceView } from '../src/core/types.ts'; +import type { Workspace } from '../src/core/workspace.ts'; + +export const HELD: InstanceView = { id: 'app_held', name: 'verify-throwaway-held', created: false, settings: 'standard' }; + +export interface FakeHeld { + readonly finish?: (ledger: Workspace, options: { readonly keepApplications: boolean }) => Promise; +} + +export function heldInstances(fake: FakeHeld = {}): Instances { + const notDriving = (): never => { + throw new Error('this test drives no run, and something asked for the applied instance'); + }; + return { + access: async () => 'a test', + recordedKey: () => null, + ensure: async () => [HELD], + apply: async () => notDriving(), + keys: notDriving, + clerk: notDriving, + stopDriving: async () => undefined, + finish: async (ledger, options) => (await fake.finish?.(ledger, options)) ?? [], + doctorChecks: async () => [], + }; +} diff --git a/integration/expo-native/testing/git-env.ts b/integration/expo-native/testing/git-env.ts new file mode 100644 index 00000000000..2a4b2900f5c --- /dev/null +++ b/integration/expo-native/testing/git-env.ts @@ -0,0 +1,19 @@ +export const REPOSITORY_LOCAL_GIT_ENV = [ + 'GIT_ALTERNATE_OBJECT_DIRECTORIES', + 'GIT_CONFIG', + 'GIT_CONFIG_PARAMETERS', + 'GIT_CONFIG_COUNT', + 'GIT_OBJECT_DIRECTORY', + 'GIT_DIR', + 'GIT_WORK_TREE', + 'GIT_IMPLICIT_WORK_TREE', + 'GIT_GRAFT_FILE', + 'GIT_INDEX_FILE', + 'GIT_NO_REPLACE_OBJECTS', + 'GIT_REPLACE_REF_BASE', + 'GIT_PREFIX', + 'GIT_SHALLOW_FILE', + 'GIT_COMMON_DIR', +] as const; + +for (const name of REPOSITORY_LOCAL_GIT_ENV) delete process.env[name]; diff --git a/integration/expo-native/testing/lock-holder.ts b/integration/expo-native/testing/lock-holder.ts new file mode 100644 index 00000000000..edb7abae707 --- /dev/null +++ b/integration/expo-native/testing/lock-holder.ts @@ -0,0 +1,12 @@ +import { appendFileSync } from 'node:fs'; +import { openWorkspace } from '../src/core/workspace.ts'; + +const [packageDir, home, log, startAt, mode] = process.argv.slice(2) as [string, string, string, string, string]; +const workspace = openWorkspace({ packageDir, worktree: packageDir, home }); +while (Date.now() < Number(startAt)) await new Promise((resolve) => setTimeout(resolve, 1)); +await workspace.withAcquireLock('ios', async () => { + if (mode === 'crash') process.kill(process.pid, 'SIGKILL'); + appendFileSync(log, `enter ${process.pid}\n`); + await new Promise((resolve) => setTimeout(resolve, 100)); + appendFileSync(log, `exit ${process.pid}\n`); +}); From 96af2c83dcb8c0e0428ac0bd9e33773a5f6aa30c Mon Sep 17 00:00:00 2001 From: Mike Pitre <12040919+mikepitre@users.noreply.github.com> Date: Wed, 7 Oct 2026 20:30:58 -0400 Subject: [PATCH 3/7] test(expo): create, configure, and delete a throwaway Clerk application per worktree Co-Authored-By: Claude Opus 5.5 --- integration/expo-native/src/core/MANIFEST | 5 + .../expo-native/src/core/instances/base.json | 298 +++++++++++ .../src/core/instances/instances.ts | 264 +++++++++ .../src/core/instances/platform.ts | 423 +++++++++++++++ .../src/core/instances/settings.ts | 261 +++++++++ .../src/core/instances/throwaway.ts | 503 ++++++++++++++++++ 6 files changed, 1754 insertions(+) create mode 100644 integration/expo-native/src/core/instances/base.json create mode 100644 integration/expo-native/src/core/instances/instances.ts create mode 100644 integration/expo-native/src/core/instances/platform.ts create mode 100644 integration/expo-native/src/core/instances/settings.ts create mode 100644 integration/expo-native/src/core/instances/throwaway.ts diff --git a/integration/expo-native/src/core/MANIFEST b/integration/expo-native/src/core/MANIFEST index 79297d099c2..fd1a27fb9d8 100644 --- a/integration/expo-native/src/core/MANIFEST +++ b/integration/expo-native/src/core/MANIFEST @@ -23,6 +23,11 @@ b59071aaefeec02dea89bd25615f8ada0da0a51e805a2523011b8b591e7435d3 src/core/e2e.t 8c0ae6c44155105ce15011176bc630c59d6b4a2d2aa050f3fad4aa9674386ad8 src/core/evidence.ts 27bfd4ce59937aa8c97d9fd6849cf52f3da05626fbc9388385f4b5e684577d36 src/core/exec.ts e2075f17f52ded8dea64d8a2473820396ca5ac1a22db59f21a092da28e55911f src/core/github-report.ts +699d3a0f0b92831da4fccfa5160aa0541e5d95029281dbef264eda2a49003850 src/core/instances/base.json +cf6aa68be73e4abaf3b63d5d84c2e9eee6197412f607ec53d0f05f7b058db420 src/core/instances/instances.ts +3f880f72c3356d67be9ab147c3d52d3dc90a388ba16dec009543a6c8cd93a9e2 src/core/instances/platform.ts +26c8c052d7a87ff5401cd3dfc61ae7b1acde0966a3675e92567b9afde0ea96c6 src/core/instances/settings.ts +7272293ca6be682c1a049ac7ee9fd54255721231b7da0fe11af5bbeb50e7e712 src/core/instances/throwaway.ts 39a849e9c07ce23e8a1be9f50a136690deba3cd22b153d66cc3f9f2f53d88576 src/core/keys.ts 72e7a64c6974afdc4612084d2e838075a9ce0edb6c676b23c266b8026400badc src/core/launch.d.mts 1287fc7e3328699e104c7bc50d3cd9bf9fbfec9efe764592c98e90e56033273d src/core/launch.mjs diff --git a/integration/expo-native/src/core/instances/base.json b/integration/expo-native/src/core/instances/base.json new file mode 100644 index 00000000000..0ee4e5c559a --- /dev/null +++ b/integration/expo-native/src/core/instances/base.json @@ -0,0 +1,298 @@ +{ + "config": { + "auth_access_control": { + "allowlist_blocklist_enforced_on_sign_in": true, + "allowlist_enabled": false, + "block_disposable_email_domains": false, + "block_email_subaddresses": false, + "blocklist_enabled": false, + "sign_up_mode": "public" + }, + "auth_attack_protection": { + "bot_protection": { + "captcha_enabled": false, + "captcha_widget_type": "smart" + }, + "email_link_require_same_client": false, + "enumeration_protection": "bulk", + "pii_protection_enabled": true, + "user_lockout": { + "duration_in_minutes": 60, + "enabled": true, + "max_attempts": 100 + } + }, + "auth_biometric": { + "enrollment_prompt_after_sign_in": false, + "enrollment_prompt_after_sign_up": false, + "used_for_sign_in": false + }, + "auth_email": { + "immutable": false, + "required_for_sign_up": true, + "sign_in_strategies": [ + "email_code", + "email_link" + ], + "used_for_sign_in": true, + "used_for_sign_up": true, + "verification_strategies": [ + "email_code" + ], + "verify_at_sign_up": true + }, + "auth_multi_factor": { + "authenticator_app": { + "enabled": true + }, + "backup_code": { + "enabled": true + }, + "required_for_sign_in": false, + "required_for_sign_up": false + }, + "auth_passkey": { + "allow_autofill": true, + "satisfies_second_factor": true, + "show_sign_in_button": true, + "used_for_sign_in": false + }, + "auth_password": { + "device_trust": { + "enabled": true + }, + "disable_hibp": false, + "disable_password_reverification": false, + "enabled": true, + "enforce_hibp_on_sign_in": false, + "max_length": 0, + "min_length": 8, + "min_zxcvbn_strength": 0, + "require_lowercase": false, + "require_numbers": false, + "require_special_char": true, + "require_uppercase": false, + "required": true, + "show_zxcvbn": false + }, + "auth_phone": { + "immutable": false, + "required_for_sign_up": false, + "second_factor_strategies": [ + "phone_code" + ], + "sign_in_strategies": [ + "phone_code" + ], + "used_for_second_factor": true, + "used_for_sign_in": true, + "used_for_sign_up": true, + "verification_strategies": [ + "phone_code" + ], + "verify_at_sign_up": true + }, + "auth_username": { + "allow_extended_special_characters": false, + "allow_numeric_usernames": false, + "immutable": false, + "max_length": 64, + "min_length": 4, + "required_for_sign_up": false, + "used_for_sign_in": true, + "used_for_sign_up": true + }, + "auth_web3": { + "required_for_sign_up": false, + "sign_in_strategies": [ + "web3_metamask_signature" + ], + "used_for_sign_in": true, + "used_for_sign_up": true, + "verification_strategies": [ + "web3_metamask_signature" + ], + "verify_at_sign_up": true + }, + "compliance": { + "legal_consent": { + "enabled": false, + "privacy_policy_url": null, + "terms_of_service_url": null + } + }, + "connection_oauth_google": { + "block_email_subaddresses": false + }, + "organization_settings": { + "admin_delete_enabled": true, + "creator_role": "org:admin", + "domains_default_role": "org:member", + "domains_enabled": true, + "domains_enrollment_modes": [ + "manual_invitation", + "automatic_invitation", + "automatic_suggestion" + ], + "enabled": true, + "force_organization_selection": false, + "initial_role_set_key": "role_set:default", + "max_allowed_domains": 10, + "max_allowed_memberships": 3, + "max_allowed_roles": 10, + "max_role_sets_allowed": 30, + "organization_creation_defaults": { + "automatic_organization_creation": { + "enabled": false + }, + "detect_from_email_domain": { + "enabled": false + }, + "enabled": false, + "fallback": { + "name": "" + }, + "organization_name_template": { + "enabled": false, + "template": "" + } + }, + "slug_disabled": false + }, + "session_settings": { + "inactivity_timeout": { + "duration_seconds": 0, + "enabled": false + }, + "maximum_lifetime": { + "duration_seconds": 604800, + "enabled": true + }, + "multi_session_enabled": true + }, + "user_model": { + "first_name": { + "enabled": true, + "required": false + }, + "last_name": { + "enabled": true, + "required": false + } + } + }, + "environment": { + "auth_config.first_factors": [ + "email_code", + "email_link", + "oauth_google", + "password", + "phone_code", + "reset_password_email_code", + "reset_password_phone_code", + "ticket", + "web3_metamask_signature" + ], + "auth_config.first_name": "on", + "auth_config.identification_requirements[0]": [ + "email_address", + "oauth_google", + "phone_number", + "web3_wallet" + ], + "auth_config.identification_requirements[1]": [ + "username" + ], + "auth_config.identification_strategies": [ + "email_address", + "oauth_google", + "phone_number", + "username", + "web3_wallet" + ], + "auth_config.last_name": "on", + "auth_config.native_settings.api_enabled": true, + "auth_config.phone_number": "on", + "auth_config.second_factors": [ + "backup_code", + "phone_code", + "totp" + ], + "auth_config.single_session_mode": false, + "auth_config.test_mode": true, + "auth_config.username": "on", + "display_config.captcha_provider": null, + "display_config.captcha_widget_type": null, + "organization_settings.domains.default_role": "org:member", + "organization_settings.domains.enabled": true, + "organization_settings.domains.enrollment_modes": [ + "automatic_invitation", + "automatic_suggestion", + "manual_invitation" + ], + "organization_settings.enabled": true, + "organization_settings.force_organization_selection": false, + "organization_settings.max_allowed_memberships": 3, + "organization_settings.organization_creation_defaults.detect_from_email_domain.enabled": false, + "organization_settings.organization_creation_defaults.enabled": false, + "organization_settings.organization_creation_defaults.fallback.name": "", + "organization_settings.organization_creation_defaults.organization_name_template.enabled": false, + "organization_settings.organization_creation_defaults.organization_name_template.template": "", + "organization_settings.slug.disabled": false, + "user_settings.actions.create_organization": true, + "user_settings.actions.delete_self": true, + "user_settings.attack_protection.email_link.require_same_client": false, + "user_settings.attack_protection.user_lockout.max_attempts": 100, + "user_settings.attributes.authenticator_app.enabled": true, + "user_settings.attributes.authenticator_app.second_factors": [ + "totp" + ], + "user_settings.attributes.authenticator_app.used_for_second_factor": true, + "user_settings.attributes.authenticator_app.verifications": [ + "totp" + ], + "user_settings.attributes.backup_code.enabled": true, + "user_settings.attributes.backup_code.second_factors": [ + "backup_code" + ], + "user_settings.attributes.backup_code.used_for_second_factor": true, + "user_settings.attributes.email_address.first_factors": [ + "email_code", + "email_link" + ], + "user_settings.attributes.first_name.enabled": true, + "user_settings.attributes.last_name.enabled": true, + "user_settings.attributes.phone_number.enabled": true, + "user_settings.attributes.phone_number.first_factors": [ + "phone_code" + ], + "user_settings.attributes.phone_number.second_factors": [ + "phone_code" + ], + "user_settings.attributes.phone_number.used_for_first_factor": true, + "user_settings.attributes.phone_number.used_for_second_factor": true, + "user_settings.attributes.phone_number.verifications": [ + "phone_code" + ], + "user_settings.attributes.phone_number.verify_at_sign_up": true, + "user_settings.attributes.ticket.enabled": true, + "user_settings.attributes.username.enabled": true, + "user_settings.attributes.username.used_for_first_factor": true, + "user_settings.attributes.web3_wallet.enabled": true, + "user_settings.attributes.web3_wallet.first_factors": [ + "web3_metamask_signature" + ], + "user_settings.attributes.web3_wallet.used_for_first_factor": true, + "user_settings.attributes.web3_wallet.verifications": [ + "web3_metamask_signature" + ], + "user_settings.attributes.web3_wallet.verify_at_sign_up": true, + "user_settings.password_settings.enforce_hibp_on_sign_in": false, + "user_settings.password_settings.min_length": 8, + "user_settings.password_settings.require_special_char": true, + "user_settings.restrictions.allowlist_blocklist_disabled_on_sign_in.enabled": false, + "user_settings.sign_up.captcha_enabled": false, + "user_settings.sign_up.mfa.required": false, + "user_settings.social.oauth_google.block_email_subaddresses": false + } +} diff --git a/integration/expo-native/src/core/instances/instances.ts b/integration/expo-native/src/core/instances/instances.ts new file mode 100644 index 00000000000..8709100b3ad --- /dev/null +++ b/integration/expo-native/src/core/instances/instances.ts @@ -0,0 +1,264 @@ +import { existsSync, mkdirSync, readdirSync } from 'node:fs'; +import { join, relative, sep } from 'node:path'; +import { BACKEND_API_HOST, createClerkBackends, type ClerkBackend } from '../clerk.ts'; +import { currentProcess, isRunning, sleep as defaultSleep, type Runner } from '../exec.ts'; +import type { InstanceKeys } from '../keys.ts'; +import { count } from '../state.ts'; +import { newEntryId, takeSlotLock, type Workspace } from '../workspace.ts'; +import { VerifyFailure, type ApplicationView, type DoctorCheck, type InstanceView, type ProcessRef } from '../types.ts'; +import { createPlatform, describeCredential, type Platform } from './platform.ts'; +import { STANDARD_FILE, declaredIn, describeDifference, readSpecText, settingsOf, straySettingsFile, type SettingsGroup } from './settings.ts'; +import { createThrowaway, openApplications, type HeldApplication, type Throwaway } from './throwaway.ts'; + +export interface AppliedInstance { + readonly keys: InstanceKeys; + readonly instance: { readonly id: string; readonly name: string }; + readonly changed: boolean; + stillApplied(): Promise; + release(): Promise; +} + +export interface Instances { + access(): Promise; + recordedKey(): string | null; + ensure(options: { readonly willChange: boolean }, progress: (line: string) => void): Promise; + apply(group: SettingsGroup, progress: (line: string) => void): Promise; + keys(): InstanceKeys; + clerk(): ClerkBackend; + stopDriving(): Promise; + finish(ledger: Workspace, options: { readonly keepApplications: boolean }, progress: (line: string) => void): Promise; + doctorChecks(options: { readonly live: boolean }, progress: (line: string) => void): Promise; +} + +export interface InstancesDeps { + readonly workspace: Workspace; + readonly env: Readonly>; + readonly runner: Runner; + readonly progress: (line: string) => void; + readonly fetch?: typeof fetch; + readonly sleep?: (ms: number) => Promise; + readonly now?: () => number; + readonly drivers?: { readonly self: ProcessRef; readonly isRunning: (driver: ProcessRef) => boolean }; +} + +export function check(id: DoctorCheck['id'], ok: boolean, detail: string, fix: string): DoctorCheck { + return ok ? { id, ok, detail } : { id, ok, detail, fix }; +} + +function specFiles(dir: string): string[] { + if (!existsSync(dir)) return []; + return readdirSync(dir, { withFileTypes: true }).flatMap((entry) => { + const path = join(dir, entry.name); + return entry.isDirectory() ? specFiles(path) : entry.name.endsWith('.e2e.ts') ? [path] : []; + }); +} + +export function createInstances(deps: InstancesDeps): Instances { + const { workspace, env } = deps; + const sleep = deps.sleep ?? defaultSleep; + const backends = createClerkBackends(deps.fetch); + const platform: Platform = createPlatform({ env, runner: deps.runner, progress: deps.progress, ...(deps.fetch === undefined ? {} : { fetch: deps.fetch }), sleep, ...(deps.now === undefined ? {} : { now: deps.now }) }); + const throwaway: Throwaway = createThrowaway({ + workspace, + platform, + clerk: backends, + env, + self: deps.drivers?.self ?? currentProcess(), + isRunning: deps.drivers?.isRunning ?? isRunning, + ...(deps.fetch === undefined ? {} : { fetch: deps.fetch }), + sleep, + ...(deps.now === undefined ? {} : { now: deps.now }), + }); + let reaching: Promise | undefined; + let applied: AppliedInstance | undefined; + + const current = (): AppliedInstance => { + if (applied === undefined) throw new VerifyFailure('NOT_READY', 'no instance is applied: a spec asked for a user or a launch outside the part of a run that drives the device', 'report this as a verify bug'); + return applied; + }; + const appliedClerk = backends(() => current().keys); + + async function reach(): Promise { + if (openApplications(workspace).length > 0) { + await platform.credential(); + return 'this worktree already holds throwaway instances'; + } + const open = await platform.open(); + return `${describeCredential(open.credential)} reaches the verification workspace ${open.workspace}`; + } + + const access = (): Promise => (reaching ??= reach()); + + async function locked(fn: () => T | Promise): Promise { + mkdirSync(join(workspace.root, 'locks'), { recursive: true }); + const release = await takeSlotLock( + join(workspace.root, 'locks', 'instances'), + Number.POSITIVE_INFINITY, + () => new VerifyFailure('NOT_READY', 'unreachable', ''), + (owner) => deps.progress(`wait another {cli} in this worktree (pid ${owner.pid}) is creating, changing, or deleting instances; waiting for it, with no time limit`), + ); + try { + return await fn(); + } finally { + release(); + } + } + + const ownKeyAccepted = async (keys: InstanceKeys): Promise => void (await backends(() => keys).userCount()); + + function declarations(): { readonly declaring: number } | { readonly refused: VerifyFailure } { + const stray = straySettingsFile(workspace.packageDir); + if (stray !== null) return { refused: stray }; + let declaring = 0; + for (const file of specFiles(join(workspace.packageDir, 'specs')).sort()) { + const path = relative(workspace.packageDir, file).split(sep).join('/'); + try { + if (settingsOf(declaredIn(readSpecText(workspace.packageDir, path), path), path).declared !== null) declaring += 1; + } catch (error) { + if (!(error instanceof VerifyFailure)) throw error; + return { refused: error }; + } + } + return { declaring }; + } + + async function settingsCheck(): Promise { + const details: string[] = []; + const fixes: string[] = []; + try { + const inspected = await throwaway.inspect(); + if (inspected === null) details.push(`none created yet; up creates one application from ${STANDARD_FILE}`); + else { + const { id, settings } = inspected.application; + const { found } = inspected; + if (found === 'gone') { + details.push(`Clerk no longer serves ${id}`); + fixes.push('{cli} up creates a new application'); + } else if (settings === null) { + details.push(`${id} has no settings recorded`); + fixes.push('{cli} up returns it to the standard settings'); + } else if (found.differing.length > 0) { + details.push(`${id} is recorded as on ${settings.label} and shows ${found.differing.slice(0, 5).map((d) => describeDifference(d, 'those settings expect')).join('; ')}`); + fixes.push('{cli} up returns it to the standard settings'); + } else { + details.push( + [ + `${id} is on ${settings.label}${settings.askedBy === null ? '' : `, which ${settings.askedBy} asked for`}`, + `${found.compared} settings match ${STANDARD_FILE}${settings.declared === null ? '' : ' with that declaration'}`, + ].join('; '), + ); + } + } + } catch (error) { + details.push(`could not read an environment: ${(error as Error).message}`); + fixes.push('check network access to *.clerk.accounts.dev'); + } + const scanned = declarations(); + if ('refused' in scanned) { + details.push(scanned.refused.message); + fixes.push(scanned.refused.fix); + } else { + details.push(scanned.declaring === 1 ? '1 spec file declares settings' : `${scanned.declaring} spec files declare settings`); + } + return check('settings', fixes.length === 0, details.join('; '), fixes.join('; ')); + } + + async function apiCheck(application: HeldApplication | null): Promise { + if (application === null) return check('clerk-api', true, `not observed yet: this worktree holds no application, so no call has used an instance's own key`, ''); + try { + await ownKeyAccepted(await application.keys()); + return check('clerk-api', true, `${BACKEND_API_HOST} accepts the own key of ${application.id}`, ''); + } catch (error) { + return check('clerk-api', false, (error as Error).message, error instanceof VerifyFailure ? error.fix : 'run `{cli} doctor` again'); + } + } + + async function ensure(options: { readonly willChange: boolean }, progress: (line: string) => void): Promise { + progress(`instances ${await access()}`); + const up = await throwaway.ensure(options, progress); + if (up.created !== null) await ownKeyAccepted(await up.created.keys()); + return up.views; + } + + async function finish(ledger: Workspace, options: { readonly keepApplications: boolean }, progress: (line: string) => void): Promise { + try { + return options.keepApplications ? [] : await throwaway.finish(ledger, progress); + } finally { + for (const entry of ledger.unclosedEntries()) { + if (entry.kind === 'user') ledger.append({ id: newEntryId(), kind: 'done', ref: entry.id }); + } + } + } + + function liveCheck(progress: (line: string) => void): Promise { + return locked(async () => { + const held = throwaway.held(); + if (held !== null) { + return [await apiCheck(held), await settingsCheck(), { id: 'live-instance', ok: true, state: 'not-run', detail: `not run: this worktree already holds ${held.id}, which the checks above read` }]; + } + const started = Date.now(); + try { + const [made] = await ensure({ willChange: false }, progress); + const api = await apiCheck(throwaway.held()); + const inspected = await settingsCheck(); + const finished = await finish(workspace, { keepApplications: false }, progress); + const what = made === undefined ? 'an application' : `${made.id} (${made.name})`; + return [api, inspected, check('live-instance', api.ok, `created ${what}, configured it, compared its environment, and deleted it (${count(finished.length, 'application')} gone from the list) in ${Math.round((Date.now() - started) / 1000)}s`, api.fix ?? '')]; + } catch (error) { + const failure = error instanceof VerifyFailure ? error : new VerifyFailure('NOT_READY', (error as Error).message, '{cli} down deletes anything it left'); + await finish(workspace, { keepApplications: false }, progress).catch(() => undefined); + return [check('live-instance', false, failure.message, failure.fix)]; + } + }); + } + + return { + access, + recordedKey: () => throwaway.held()?.settings?.key ?? null, + keys: () => current().keys, + clerk: () => { + current(); + return appliedClerk; + }, + ensure: (options, progress) => locked(() => ensure(options, progress)), + + async apply(group, progress) { + if (applied !== undefined) throw new Error('an instance is still applied: release it before applying the next group'); + const application = await locked(() => throwaway.apply(group, progress)); + applied = { + keys: application.keys, + instance: { id: application.id, name: application.name }, + changed: application.changed, + stillApplied: application.stillApplied, + release: async () => { + applied = undefined; + }, + }; + return applied; + }, + + stopDriving: () => locked(throwaway.stopDriving), + + finish: (ledger, options, progress) => (ledger.root === workspace.root ? locked(() => finish(ledger, options, progress)) : finish(ledger, options, progress)), + + async doctorChecks(options, progress) { + const failed = (error: unknown, prefix = ''): DoctorCheck => { + const failure = error instanceof VerifyFailure ? error : new VerifyFailure('NOT_READY', (error as Error).message, 'run `{cli} doctor` again'); + return check('instances', false, `${prefix}${failure.message}`, failure.fix); + }; + const held = throwaway.held(); + const holding = held === null ? 'none created yet' : `${held.id} (${held.name}) on ${held.settings?.label ?? 'unknown settings'}`; + let reaches: DoctorCheck; + try { + await platform.requireScopes(); + const open = await platform.open(); + reaches = check('instances', true, `${describeCredential(open.credential)} reaches the verification workspace ${open.workspace}; ${holding}`, ''); + } catch (error) { + if (openApplications(workspace).length === 0) return [failed(error)]; + reaches = failed(error, `${holding}; `); + } + if (options.live && reaches.ok) return [reaches, ...(await liveCheck(progress))]; + return [reaches, await apiCheck(held), await settingsCheck()]; + }, + }; +} diff --git a/integration/expo-native/src/core/instances/platform.ts b/integration/expo-native/src/core/instances/platform.ts new file mode 100644 index 00000000000..34104399d64 --- /dev/null +++ b/integration/expo-native/src/core/instances/platform.ts @@ -0,0 +1,423 @@ +import { readFileSync, statSync } from 'node:fs'; +import { sleep as defaultSleep, type Runner } from '../exec.ts'; +import { Secret } from '../../../specs/support/secret.ts'; +import { VerifyFailure, type Brand, type Json, type PublishableKey } from '../types.ts'; +import { SettingsRefused } from './settings.ts'; + +export const THROWAWAY_PREFIX = 'verify-throwaway-'; + +const DEADLINE = /^verify-throwaway-until-(\d{4})(\d{2})(\d{2})t(\d{2})(\d{2})z-[0-9a-f]{8}$/; + +export function throwawayName(deadline: Date, random: string): string { + const stamp = deadline.toISOString().replace(/[-:]/g, '').slice(0, 13).toLowerCase(); + return `${THROWAWAY_PREFIX}until-${stamp}z-${random}`; +} + +export function deadlineOf(name: string): Date | null { + const match = DEADLINE.exec(name); + if (match === null) return null; + const [, year, month, day, hour, minute] = match; + return new Date(Date.UTC(Number(year), Number(month) - 1, Number(day), Number(hour), Number(minute))); +} + +const VERIFICATION_WORKSPACE = 'org_3KHungJxbvIscuSvy8oos5MHAli'; + +const REFERENCE_SHAPE = 'op:////credential'; +const REFERENCED_ITEM = 'the 1Password item the reference names'; + +const PLATFORM_API = 'https://api.clerk.com/v1/platform'; +const SECRET_KEY_SCOPE = 'application_secret_keys:read'; +const NEEDED_SCOPES = ['applications:read', 'applications:manage', 'applications:delete', SECRET_KEY_SCOPE] as const; +const listed = (names: readonly string[]): string => new Intl.ListFormat('en', { type: 'conjunction' }).format(names); +const scopesNamed = (scopes: readonly string[]): string => `the scope${scopes.length === 1 ? '' : 's'} ${listed(scopes)}`; +const OP_WAIT_SECONDS = 60; +const GUARD_FRESH_MS = 30_000; +const RATE_LIMIT_WAITS = [2, 4, 8, 16, 30, 30] as const; +const RETRY_AFTER_CAP_SECONDS = 60; +const NOT_ABOUT_THE_BODY: ReadonlySet = new Set([401, 403, 404, 408]); + +export type PlatformCredential = + | { readonly via: 'environment'; readonly variable: 'CLERK_PLATFORM_API_KEY' | 'CLERK_PLATFORM_API_KEY_FILE'; readonly key: Secret<'clerk-platform-key'> } + | { readonly via: 'one-password'; readonly key: Secret<'clerk-platform-key'> } + | { readonly via: 'proxy' }; + +export function describeCredential(credential: PlatformCredential): string { + switch (credential.via) { + case 'environment': + return credential.variable; + case 'one-password': + return '1Password'; + case 'proxy': + return 'a key attached outside this machine (no key is in this process)'; + default: { + const exhaustive: never = credential; + return exhaustive; + } + } +} + +export type ThrowawayApplication = Brand<{ readonly id: string; readonly name: string }, 'ThrowawayApplication'>; + +export function throwawayApplication(id: string, name: string): ThrowawayApplication { + if (!name.startsWith(THROWAWAY_PREFIX)) throw new Error(`${name} lacks the ${THROWAWAY_PREFIX} prefix`); + return { id, name } as ThrowawayApplication; +} + +export interface CreatedApplication { + readonly application: ThrowawayApplication; + readonly instanceId: string; + readonly pk: PublishableKey; + readonly sk: Secret<'clerk-secret-key'>; +} + +export interface Listing { + readonly applications: readonly ThrowawayApplication[]; + readonly at: Date | null; +} + +export interface OpenWorkspace { + readonly credential: PlatformCredential; + readonly workspace: string; + readonly opened: Listing; + list(): Promise; + create(name: string): Promise; + configure(created: Pick, config: { readonly [key: string]: Json }, options?: { readonly dryRun?: boolean }): Promise<{ readonly after: Json }>; + delete(application: ThrowawayApplication): Promise; +} + +export interface Platform { + open(): Promise; + credential(): Promise; + requireScopes(): Promise; + secretKey(application: ThrowawayApplication): Promise>; +} + +export interface PlatformDeps { + readonly env: Readonly>; + readonly runner: Runner; + readonly progress: (line: string) => void; + readonly fetch?: typeof fetch; + readonly sleep?: (ms: number) => Promise; + readonly now?: () => number; +} + +interface Answer { + readonly status: number; + readonly json: unknown; + readonly codes: readonly string[]; + readonly missingScopes: readonly string[]; + readonly date: Date | null; +} + +interface Resolved { + readonly credential: PlatformCredential; + readonly scopes: readonly string[]; +} + +type Key = Secret<'clerk-platform-key'> | null; + +const strings = (value: unknown): readonly string[] => (Array.isArray(value) ? value.filter((each): each is string => typeof each === 'string') : []); + +const noCredential = (tried: readonly string[], onAMac: string): VerifyFailure => + new VerifyFailure( + 'KEYS_MISSING', + `no Clerk Platform API credential works here (${tried.join('; ')})`, + `on a Mac, ${onAMac}; in a cloud environment, add an API credential for api.clerk.com with path prefix /v1/platform/; anywhere, set CLERK_PLATFORM_API_KEY to the team key, or CLERK_PLATFORM_API_KEY_FILE to a file that holds it and that only you can read`, + ); + +type KeyReference = Secret<'one-password-reference'>; + +function keyReference(env: PlatformDeps['env']): KeyReference | null { + const held = (value: string, source: string): KeyReference => { + if (!value.startsWith('op://')) throw new VerifyFailure('USAGE', `${source} does not hold a 1Password secret reference`, `put a reference of the shape ${REFERENCE_SHAPE} there, or remove it`); + return new Secret('one-password-reference', value); + }; + const inline = env.VERIFY_PLATFORM_KEY_REFERENCE?.trim() ?? ''; + return inline === '' ? null : held(inline, 'VERIFY_PLATFORM_KEY_REFERENCE'); +} + +function withoutReference(text: string, reference: string): string { + const named = reference.slice('op://'.length).split('/').slice(0, 2); + return [reference, ...named].filter((part) => part !== '').reduce((out, part) => out.split(part).join(''), text); +} + +function keyFromText(text: string, source: string): Secret<'clerk-platform-key'> { + const value = text.trim(); + if (!/^ak_[A-Za-z0-9_-]+$/.test(value)) throw new VerifyFailure('KEYS_MISSING', `${source} does not hold a Clerk Platform API key (those start with ak_)`, `put the team key in ${source}, or unset it`); + return new Secret('clerk-platform-key', value); +} + +function environmentCredential(env: PlatformDeps['env']): Extract | null { + const inline = env.CLERK_PLATFORM_API_KEY; + if (inline !== undefined && inline.trim() !== '') return { via: 'environment', variable: 'CLERK_PLATFORM_API_KEY', key: keyFromText(inline, 'CLERK_PLATFORM_API_KEY') }; + const file = env.CLERK_PLATFORM_API_KEY_FILE; + if (file === undefined || file.trim() === '') return null; + let mode: number; + try { + mode = statSync(file).mode; + } catch { + throw new VerifyFailure('KEYS_MISSING', `CLERK_PLATFORM_API_KEY_FILE names ${file}, which does not exist`, 'unset CLERK_PLATFORM_API_KEY_FILE, or point it at a file that holds the key'); + } + if ((mode & 0o077) !== 0) throw new VerifyFailure('KEYS_MISSING', `${file} can be read by other users of this machine`, `chmod 600 ${file}`); + return { via: 'environment', variable: 'CLERK_PLATFORM_API_KEY_FILE', key: keyFromText(readFileSync(file, 'utf8'), file) }; +} + +export function createPlatform(deps: PlatformDeps): Platform { + const request = deps.fetch ?? fetch; + const sleep = deps.sleep ?? defaultSleep; + const now = deps.now ?? Date.now; + let opened: Promise | undefined; + let resolved: Promise | undefined; + + async function send(key: Key, method: string, path: string, body?: unknown): Promise { + for (let attempt = 0; ; attempt += 1) { + const headers: Record = { 'User-Agent': 'verify-instances' }; + if (body !== undefined) headers['Content-Type'] = 'application/json'; + const call = (authorization?: string) => + request(`${PLATFORM_API}${path}`, { + method, + headers: authorization === undefined ? headers : { ...headers, Authorization: authorization }, + ...(body === undefined ? {} : { body: JSON.stringify(body) }), + signal: AbortSignal.timeout(30_000), + }); + let response: Response; + try { + response = await (key === null ? call() : key.use('platform-authorization', (plain) => call(`Bearer ${plain}`))); + } catch (error) { + const cause = (error as { cause?: { code?: string } }).cause?.code; + throw new VerifyFailure('NOT_READY', `Clerk's Platform API did not answer ${method} ${path}: ${(error as Error).message}${cause === undefined ? '' : ` (${cause})`}`, 'check network access to api.clerk.com; a cloud environment needs it in its allowed domains'); + } + const wait = RATE_LIMIT_WAITS[attempt]; + if (response.status === 429 && wait !== undefined) { + const asked = Number(response.headers.get('retry-after')); + const seconds = Number.isFinite(asked) && asked > 0 ? Math.min(asked, RETRY_AFTER_CAP_SECONDS) : wait; + deps.progress(`wait Clerk's Platform API is rate limiting the verification workspace (100 requests a minute, shared by every session); retrying ${method} ${path} in ${seconds}s`); + await response.arrayBuffer().catch(() => undefined); + await sleep(seconds * 1000); + continue; + } + const text = await response.text(); + let json: unknown = null; + try { + json = text === '' ? null : JSON.parse(text); + } catch { + json = null; + } + const found = (json as { errors?: { code?: unknown; meta?: { scopes?: unknown } | null }[] } | null)?.errors; + const errors = Array.isArray(found) ? found : []; + const date = Date.parse(response.headers.get('date') ?? ''); + return { + status: response.status, + json, + codes: errors.map((e) => (typeof e.code === 'string' ? e.code : 'unknown')), + missingScopes: errors.flatMap((e) => strings(e.meta?.scopes)), + date: Number.isNaN(date) ? null : new Date(date), + }; + } + } + + const said = (answer: Answer): string => `${answer.status}${answer.codes.length === 0 ? '' : ` ${[...new Set(answer.codes)].join(', ')}`}`; + + function refused(answer: Answer, what: string): VerifyFailure { + if (answer.status === 429) return new VerifyFailure('RATE_LIMITED', `Clerk's Platform API kept rate limiting ${what}`, 'wait a minute and rerun; the limit is 100 requests a minute for the whole verification workspace'); + if (answer.codes.includes('authorization_missing_scopes')) return new VerifyFailure('KEYS_MISSING', `the Platform API key lacks a scope that ${what} needs (${said(answer)})`, `use a key with ${listed(NEEDED_SCOPES)}`); + return new VerifyFailure('NOT_READY', `Clerk's Platform API answered ${said(answer)} to ${what}`, 'rerun; if it repeats, run `{cli} doctor`'); + } + + async function reaches(key: Key): Promise<{ readonly workspace: string; readonly scopes: readonly string[] } | { readonly refusal: string }> { + const answer = await send(key, 'GET', '/me'); + if (answer.status === 401 || answer.status === 403) return { refusal: said(answer) }; + const principal = answer.json as { subject?: unknown; scopes?: unknown } | null; + if (answer.status !== 200 || typeof principal?.subject !== 'string') throw refused(answer, 'the request that asks which workspace the key belongs to'); + return { workspace: principal.subject, scopes: strings(principal.scopes) }; + } + + const keyOf = (credential: PlatformCredential): string => (credential.via === 'one-password' ? `the key in ${REFERENCED_ITEM}` : describeCredential(credential)); + + function wrongWorkspace(found: string, credential: PlatformCredential): VerifyFailure { + return new VerifyFailure( + 'KEYS_MISSING', + `${keyOf(credential)} belongs to workspace ${found}, and verification creates applications only in ${VERIFICATION_WORKSPACE}`, + 'use the team key; a key of any other workspace is refused', + ); + } + + async function resolve(): Promise { + let reference: KeyReference | null = null; + let unreadable: unknown; + try { + reference = keyReference(deps.env); + } catch (error) { + unreadable = error; + } + const checked = async (credential: PlatformCredential, key: Key, onRefusal: (refusal: string) => VerifyFailure): Promise => { + const reached = await reaches(key); + if ('refusal' in reached) throw onRefusal(reached.refusal); + if (reached.workspace !== VERIFICATION_WORKSPACE) throw wrongWorkspace(reached.workspace, credential); + return { credential, scopes: reached.scopes }; + }; + + const fromEnvironment = environmentCredential(deps.env); + if (fromEnvironment !== null) { + return checked(fromEnvironment, fromEnvironment.key, (refusal) => new VerifyFailure('KEYS_MISSING', `${fromEnvironment.variable} is set, and Clerk's Platform API answered ${refusal} to it`, 'set it to the team key, or unset it so the next source is tried')); + } + const tried = ['CLERK_PLATFORM_API_KEY and CLERK_PLATFORM_API_KEY_FILE are not set']; + + const bare = await reaches(null); + if ('workspace' in bare) { + const credential: PlatformCredential = { via: 'proxy' }; + if (bare.workspace !== VERIFICATION_WORKSPACE) throw wrongWorkspace(bare.workspace, credential); + return { credential, scopes: bare.scopes }; + } + tried.push(`a request with no key gets ${bare.refusal}, so nothing outside this machine attaches one`); + + if (unreadable !== undefined) throw unreadable; + if (reference === null) { + tried.push('no 1Password reference is set'); + throw noCredential( + tried, + `with the 1Password CLI installed and its desktop app integration on, set VERIFY_PLATFORM_KEY_REFERENCE to the 1Password secret reference of the item that holds the team key, of the shape ${REFERENCE_SHAPE}`, + ); + } + if ((await deps.runner('op', ['--version'])).code !== 0) { + tried.push('the 1Password CLI (op) is not installed'); + throw noCredential(tried, 'install the 1Password CLI and turn on its desktop app integration'); + } + deps.progress(`wait reading the team key from 1Password; approve the request in the 1Password app within ${OP_WAIT_SECONDS}s`); + const read = await reference.use('one-password-read', async (plain) => { + const result = await deps.runner('op', ['read', '--no-newline', plain], { timeoutMs: OP_WAIT_SECONDS * 1000 }); + return { ...result, stderr: withoutReference(result.stderr, plain) }; + }); + if (read.code !== 0 || read.stdout.trim() === '') { + const why = read.code === 124 ? `was not approved within ${OP_WAIT_SECONDS}s` : `failed: ${read.stderr.trim().split('\n')[0] || `exit ${read.code}`}`; + throw new VerifyFailure('KEYS_MISSING', `op read of ${REFERENCED_ITEM} ${why}`, 'approve the request in the 1Password app (Settings, Developer, "Integrate with 1Password CLI" must be on) and rerun; or set CLERK_PLATFORM_API_KEY'); + } + const credential: PlatformCredential = { via: 'one-password', key: keyFromText(read.stdout, REFERENCED_ITEM) }; + return checked(credential, credential.key, (refusal) => new VerifyFailure('KEYS_MISSING', `Clerk's Platform API answered ${refusal} to the key in ${REFERENCED_ITEM}`, 'the item holds a key that no longer works; replace it with a current team key')); + } + + function parseEntries(json: unknown): readonly { readonly id: string; readonly name: string }[] | null { + if (!Array.isArray(json)) return null; + const out: { id: string; name: string }[] = []; + for (const item of json as readonly { application_id?: unknown; name?: unknown }[]) { + if (typeof item?.application_id !== 'string' || typeof item.name !== 'string') return null; + out.push({ id: item.application_id, name: item.name }); + } + return out; + } + + async function everyApplication(key: Key): Promise<{ readonly entries: readonly { readonly id: string; readonly name: string }[]; readonly date: Date | null }> { + const answer = await send(key, 'GET', '/applications'); + if (answer.status !== 200) throw refused(answer, 'the application list'); + const entries = parseEntries(answer.json); + if (entries === null) { + throw new VerifyFailure('NOT_READY', "Clerk's application list no longer has a shape this tool can read in full, so it cannot tell what the workspace holds", 'report this as a verify bug; nothing was created or deleted'); + } + return { entries, date: answer.date }; + } + + const resolvedOnce = (): Promise => (resolved ??= resolve()); + const credentialOnce = async (): Promise => (await resolvedOnce()).credential; + const keyIn = (credential: PlatformCredential): Key => (credential.via === 'proxy' ? null : credential.key); + + async function requireScopes(): Promise { + const { scopes } = await resolvedOnce(); + const missing = NEEDED_SCOPES.filter((scope) => !scopes.includes(scope)); + if (missing.length > 0) throw new VerifyFailure('KEYS_MISSING', `the Platform API key lacks ${scopesNamed(missing)}, which verification needs`, `add ${listed(missing)} to the Platform API key`); + } + + async function secretKey(application: ThrowawayApplication): Promise> { + const what = `reading the secret key of ${application.name}`; + const answer = await send(keyIn(await credentialOnce()), 'GET', `/applications/${application.id}?include_secret_keys=true`); + if (answer.codes.includes('authorization_missing_scopes')) { + const missing = answer.missingScopes.length === 0 ? [SECRET_KEY_SCOPE] : answer.missingScopes; + throw new VerifyFailure( + 'KEYS_MISSING', + `the Platform API key lacks ${scopesNamed(missing)}, which ${what} needs (${said(answer)})`, + `add ${listed(missing)} to the Platform API key; the secret key of an application is kept on no disk, so each command that needs it reads it from Clerk`, + ); + } + if (answer.status !== 200) throw refused(answer, what); + const instances = (answer.json as { instances?: readonly { environment_type?: unknown; secret_key?: unknown }[] } | null)?.instances; + const development = Array.isArray(instances) ? instances.find((instance) => instance.environment_type === 'development') : undefined; + if (typeof development?.secret_key !== 'string') { + throw new VerifyFailure('NOT_READY', `Clerk's answer about ${application.name} carries no secret key of a development instance`, 'rerun; if it repeats, report this as a verify bug'); + } + return new Secret('clerk-secret-key', development.secret_key); + } + + async function open(): Promise { + const credential = await credentialOnce(); + const key = keyIn(credential); + let checkedAt = 0; + + async function list(): Promise { + const { entries, date } = await everyApplication(key); + const foreign = entries.filter((entry) => !entry.name.startsWith(THROWAWAY_PREFIX)); + if (foreign.length > 0) { + throw new VerifyFailure( + 'NOT_READY', + `workspace ${VERIFICATION_WORKSPACE} holds ${foreign.length} application(s) whose name does not start with ${THROWAWAY_PREFIX} (${foreign.map((entry) => entry.id).join(', ')}), so nothing is created or deleted there`, + 'the verification workspace must hold only throwaway applications; move those out of it', + ); + } + checkedAt = now(); + return { applications: entries.map((entry) => throwawayApplication(entry.id, entry.name)), at: date }; + } + + const guard = async (): Promise => { + if (now() - checkedAt <= GUARD_FRESH_MS) return; + const reached = await reaches(key); + if ('refusal' in reached) throw new VerifyFailure('KEYS_MISSING', `Clerk's Platform API answered ${reached.refusal} to ${keyOf(credential)}`, 'run `{cli} doctor`'); + if (reached.workspace !== VERIFICATION_WORKSPACE) throw wrongWorkspace(reached.workspace, credential); + await list(); + }; + + return { + credential, + workspace: VERIFICATION_WORKSPACE, + opened: await list(), + list, + async create(name) { + throwawayApplication('', name); + await guard(); + const answer = await send(key, 'POST', '/applications', { name }); + const body = answer.json as { application_id?: unknown; instances?: readonly { environment_type?: unknown; instance_id?: unknown; publishable_key?: unknown; secret_key?: unknown }[] } | null; + if (answer.status !== 200 && answer.status !== 201) throw refused(answer, `creating ${name}`); + const development = body?.instances?.find((instance) => instance.environment_type === 'development'); + if (typeof body?.application_id !== 'string' || typeof development?.instance_id !== 'string' || typeof development.publishable_key !== 'string' || typeof development.secret_key !== 'string') { + throw new VerifyFailure('NOT_READY', `Clerk created ${name} and the answer carries no development instance keys`, '{cli} down deletes it; then report this as a verify bug'); + } + return { + application: throwawayApplication(body.application_id, name), + instanceId: development.instance_id, + pk: development.publishable_key as PublishableKey, + sk: new Secret('clerk-secret-key', development.secret_key), + }; + }, + async configure(created, config, options = {}) { + const answer = await send(key, 'PATCH', `/applications/${created.application.id}/instances/${created.instanceId}/config${options.dryRun === true ? '?dry_run=true' : ''}`, config); + if (answer.status === 200) return { after: (answer.json as { after?: Json } | null)?.after ?? null }; + const first = (answer.json as { errors?: readonly { message?: unknown; long_message?: unknown; meta?: { param_name?: unknown } }[] } | null)?.errors?.[0]; + const param = typeof first?.meta?.param_name === 'string' ? first.meta.param_name : null; + if (answer.status < 400 || answer.status >= 500 || answer.status === 429 || (param === null && NOT_ABOUT_THE_BODY.has(answer.status))) throw refused(answer, `configuring ${created.application.name}`); + const message = typeof first?.long_message === 'string' ? first.long_message : typeof first?.message === 'string' ? first.message : null; + const what = param === null ? said(answer) : `${param} (${said(answer)})`; + const refusal = { param, said: message === null ? what : `${what}: ${message}` }; + throw new SettingsRefused(`Clerk's Platform API refused the config of ${created.application.name}: ${refusal.said}`, 'run `{cli} doctor`', refusal); + }, + async delete(application) { + await guard(); + const answer = await send(key, 'DELETE', `/applications/${application.id}`); + if (answer.status !== 200 && answer.status !== 404) throw refused(answer, `deleting ${application.name}`); + }, + }; + } + + return { + open: () => (opened ??= open()), + credential: credentialOnce, + requireScopes, + secretKey, + }; +} diff --git a/integration/expo-native/src/core/instances/settings.ts b/integration/expo-native/src/core/instances/settings.ts new file mode 100644 index 00000000000..35d946d5638 --- /dev/null +++ b/integration/expo-native/src/core/instances/settings.ts @@ -0,0 +1,261 @@ +import { createHash } from 'node:crypto'; +import { existsSync, readFileSync, readdirSync } from 'node:fs'; +import { dirname, join, sep } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { VerifyFailure, type InstanceSettings, type Json, type SpecRef } from '../types.ts'; + +export type Leaves = Readonly>; + +type JsonObject = { readonly [key: string]: Json }; + +export interface StandardFile { + readonly config: JsonObject; + readonly environment: Leaves; +} + +export const STANDARD_FILE = 'src/core/instances/base.json'; + +const standard = JSON.parse(readFileSync(join(dirname(fileURLToPath(import.meta.url)), 'base.json'), 'utf8')) as StandardFile; + +export const standardFile = (): StandardFile => standard; + +export interface Settings { + readonly key: string; + readonly label: string; + readonly declared: InstanceSettings | null; + readonly askedBy: string | null; +} + +export interface PlannedSpec extends SpecRef { + readonly sourceHash: string; +} + +export interface SettingsGroup { + readonly settings: Settings; + readonly specs: readonly PlannedSpec[]; +} + +export class SettingsRefused extends VerifyFailure { + readonly param: string | null; + readonly said: string; + constructor(message: string, fix: string, refusal: { readonly param: string | null; readonly said: string } = { param: null, said: message }) { + super('INSTANCE_MISCONFIGURED', message, fix); + this.param = refusal.param; + this.said = refusal.said; + } +} + +const isObject = (value: Json | undefined): value is JsonObject => typeof value === 'object' && value !== null && !Array.isArray(value); + +export function flattenEnvironment(value: Json, path = '', out: Record = {}): Leaves { + if (Array.isArray(value)) { + if (value.every((item) => typeof item !== 'object' || item === null)) out[path] = value.map(String).sort(); + else value.forEach((item, index) => flattenEnvironment(item, `${path}[${index}]`, out)); + } else if (isObject(value)) { + for (const [key, child] of Object.entries(value)) flattenEnvironment(child, path === '' ? key : `${path}.${key}`, out); + } else { + out[path] = value; + } + return out; +} + +export interface EnvironmentDifference { + readonly path: string; + readonly expected: Json; + readonly found: Json | undefined; +} + +export interface EnvironmentComparison { + readonly compared: number; + readonly differing: readonly EnvironmentDifference[]; +} + +export function compareEnvironment(expected: Leaves, live: Json): EnvironmentComparison { + const shown = flattenEnvironment(live); + const differing = Object.entries(expected).flatMap(([path, want]): EnvironmentDifference[] => { + const found = Object.hasOwn(shown, path) ? shown[path] : undefined; + return JSON.stringify(found) === JSON.stringify(want) ? [] : [{ path, expected: want, found }]; + }); + return { compared: Object.keys(expected).length, differing }; +} + +export const describeDifference = (d: EnvironmentDifference, expectedBy = 'the file says'): string => `${d.path} is ${d.found === undefined ? 'absent' : JSON.stringify(d.found)}, and ${expectedBy} ${JSON.stringify(d.expected)}`; + +function canonical(value: Json): string { + if (Array.isArray(value)) return `[${value.map(canonical).join(',')}]`; + if (isObject(value)) return `{${Object.keys(value).sort().map((key) => `${JSON.stringify(key)}:${canonical(value[key]!)}`).join(',')}}`; + return JSON.stringify(value); +} + +function merge(base: JsonObject, over: JsonObject): JsonObject { + const out: Record = { ...base }; + for (const [key, value] of Object.entries(over)) { + const under = out[key]; + out[key] = isObject(value) && isObject(under) ? merge(under, value) : value; + } + return out; +} + +export function configLeaves(value: Json, path = '', out: Record = {}): Leaves { + if (isObject(value) && Object.keys(value).length > 0) { + for (const [key, child] of Object.entries(value)) configLeaves(child, path === '' ? key : `${path}.${key}`, out); + } else { + out[path] = value; + } + return out; +} + +const unordered = (value: Json): Json => (Array.isArray(value) ? value.map(canonical).sort() : value); + +export const sameLeaf = (a: Json | undefined, b: Json | undefined): boolean => a !== undefined && b !== undefined && canonical(unordered(a)) === canonical(unordered(b)); + +const keyOf = (config: JsonObject): string => createHash('sha256').update(canonical(config)).digest('hex').slice(0, 12); + +export const STANDARD: Settings = { key: keyOf(standard.config), label: 'standard', declared: null, askedBy: null }; + +export const STANDARD_ENVIRONMENT_KEY = keyOf(standard.environment); + +export const settingsFileOf = (specPath: string): string => specPath.replace(/\.e2e\.ts$/, '.settings.json'); + +export interface SpecText { + readonly source: string; + readonly declaration: string | null; +} + +export function readSpecText(packageDir: string, specPath: string): SpecText { + const file = join(packageDir, settingsFileOf(specPath)); + return { source: readFileSync(join(packageDir, specPath), 'utf8'), declaration: existsSync(file) ? readFileSync(file, 'utf8') : null }; +} + +export const sourceHash = (text: SpecText): string => createHash('sha256').update(JSON.stringify([text.source, text.declaration])).digest('hex'); + +const FORM = '{ "config": { "auth_multi_factor": { "required_for_sign_up": true } }, "environment": { "user_settings.sign_up.mfa.required": true } }'; + +export function straySettingsFile(packageDir: string): VerifyFailure | null { + const specs = join(packageDir, 'specs'); + if (!existsSync(specs)) return null; + const files = (readdirSync(specs, { recursive: true }) as string[]).map((name) => name.split(sep).join('/')).filter((name) => /\.jsonc?$/.test(name)); + const stray = files.sort().find((name) => !name.endsWith('.settings.json') || !existsSync(join(specs, name.replace(/\.settings\.json$/, '.e2e.ts')))); + if (stray === undefined) return null; + return new VerifyFailure('USAGE', `specs/${stray} is not the settings file of a spec, so no run reads it`, 'name a settings file after its spec file, with .settings.json in place of .e2e.ts, or delete it'); +} + +export function declaredIn(text: SpecText, specPath: string): InstanceSettings | null { + const file = settingsFileOf(specPath); + if (text.declaration === null) return null; + const fail: (why: string) => never = (why) => { + throw new VerifyFailure('USAGE', `${file}: ${why}`, `write the settings of ${specPath} as one JSON object, as in \`${FORM}\``); + }; + let declared: Json; + try { + declared = JSON.parse(text.declaration) as Json; + } catch (error) { + return fail(`it is not JSON (${(error as Error).message})`); + } + if (!isObject(declared)) return fail('it is not a JSON object'); + const extra = Object.keys(declared).filter((name) => name !== 'config' && name !== 'environment'); + if (extra.length > 0) fail(`it has ${extra.join(', ')}, and only config and environment are read`); + const { config, environment } = declared; + if (!isObject(config) || Object.keys(config).length === 0) return fail('it needs a config object with the Platform API settings to change'); + if (!isObject(environment) || Object.keys(environment).length === 0) { + return fail('it needs an environment object with at least one leaf of the instance\'s public environment that shows the change'); + } + for (const [leaf, value] of Object.entries(environment)) { + const scalar = (item: Json): boolean => typeof item !== 'object' || item === null; + if (!(scalar(value) || (Array.isArray(value) && value.every(scalar)))) fail(`environment leaf ${leaf} is an object; write each leaf by its full dotted path, as in "user_settings.sign_up.mfa.required": true`); + } + return { config, environment }; +} + +const where = (askedBy: string | null): string => (askedBy === null ? 'the declaration' : askedBy); + +const KEYS_LISTED = 12; +const listed = (keys: readonly string[]): string => `${keys.slice(0, KEYS_LISTED).join(', ')}${keys.length > KEYS_LISTED ? `, and ${keys.length - KEYS_LISTED} more` : ''}`; + +function unpinned(leaf: string, value: Json, askedBy: string | null): VerifyFailure { + const segments = leaf.split('.'); + let found: Json = standard.config; + let depth = 0; + while (depth < segments.length && isObject(found) && Object.hasOwn(found, segments[depth]!)) { + found = found[segments[depth]!]!; + depth += 1; + } + if (depth === segments.length && isObject(found)) { + return new VerifyFailure( + 'INSTANCE_MISCONFIGURED', + `${where(askedBy)} declares ${leaf} as ${JSON.stringify(value)}, and the standard file has an object there`, + `declare the keys of \`${leaf}\` to change, each by its own name; the standard file has ${listed(Object.keys(found))} there`, + ); + } + const beside = isObject(found) ? `the standard file has ${listed(Object.keys(found))} ${depth === 0 ? 'at the top of `config`' : `under \`${segments.slice(0, depth).join('.')}\``}; ` : ''; + return new VerifyFailure( + 'INSTANCE_MISCONFIGURED', + `${where(askedBy)} declares ${leaf}, and the standard file has no value to return it to`, + `check the spelling against Clerk's Platform API config; ${beside}add the standard value of \`${leaf}\` to \`config\` in ${STANDARD_FILE}; a setting the Platform API config has no key for cannot be declared`, + ); +} + +export function settingsOf(declared: InstanceSettings | null, askedBy: string | null): Settings { + if (declared === null) return STANDARD; + const pinned = configLeaves(standard.config); + const leaves = configLeaves(declared.config); + for (const [leaf, value] of Object.entries(leaves)) { + if (!Object.hasOwn(pinned, leaf)) throw unpinned(leaf, value, askedBy); + } + const body = merge(standard.config, declared.config); + if (canonical(body) === canonical(standard.config)) { + throw new VerifyFailure('USAGE', `${where(askedBy)} declares only standard values (${Object.keys(leaves).join(', ')})`, `delete ${askedBy === null ? 'the settings file' : settingsFileOf(askedBy)}; a spec with none runs on the standard settings`); + } + for (const [leaf, value] of Object.entries(declared.environment)) { + if (sameLeaf(standard.environment[leaf], value)) { + throw new VerifyFailure( + 'USAGE', + `${where(askedBy)} expects ${leaf} to be ${JSON.stringify(value)}, which is its standard value, so it cannot show that the change took effect`, + 'declare a leaf the setting changes, with the value it changes to', + ); + } + } + const label = Object.entries(leaves).map(([leaf, value]) => `${leaf}=${JSON.stringify(value)}`).join(', '); + return { key: keyOf(body), label, declared, askedBy }; +} + +export function configFor(settings: Settings): JsonObject { + return settings.declared === null ? standard.config : merge(standard.config, settings.declared.config); +} + +export function expectedEnvironment(settings: Settings): Leaves { + const declared = Object.fromEntries(Object.entries(settings.declared?.environment ?? {}).map(([leaf, value]) => [leaf, Array.isArray(value) ? value.map(String).sort() : value])); + return { ...standard.environment, ...declared }; +} + +export function planGroups(specs: readonly ({ readonly spec: SpecRef } & SpecText)[], applied: string | null): readonly SettingsGroup[] { + const groups = new Map }>(); + for (const { spec, ...text } of specs) { + const settings = settingsOf(declaredIn(text, spec.path), spec.path); + const planned: PlannedSpec = { ...spec, sourceHash: sourceHash(text) }; + const group = groups.get(settings.key); + if (group === undefined) { + groups.set(settings.key, { settings, specs: [planned], leafFrom: new Map(Object.keys(settings.declared?.environment ?? {}).map((leaf) => [leaf, spec.path])) }); + continue; + } + group.specs.push(planned); + if (settings.declared === null || group.settings.declared === null) continue; + const environment: Record = { ...group.settings.declared.environment }; + for (const [leaf, value] of Object.entries(settings.declared.environment)) { + const first = group.leafFrom.get(leaf); + if (first !== undefined && !sameLeaf(environment[leaf], value)) { + throw new VerifyFailure( + 'USAGE', + `${first} and ${spec.path} declare the same config and expect different values of ${leaf} (${JSON.stringify(environment[leaf])} and ${JSON.stringify(value)})`, + 'one config shows one environment: make the two declarations expect the same value', + ); + } + if (first === undefined) group.leafFrom.set(leaf, spec.path); + environment[leaf] = value; + } + group.settings = { ...group.settings, declared: { config: group.settings.declared.config, environment } }; + } + const ordered = [...groups.values()].map(({ settings, specs: files }): SettingsGroup => ({ settings, specs: files })); + const rank = (group: SettingsGroup): number => (group.settings.key === applied ? 0 : group.settings.key === STANDARD.key ? 2 : 1); + return ordered.map((group, index) => ({ group, index })).sort((a, b) => rank(a.group) - rank(b.group) || a.index - b.index).map(({ group }) => group); +} diff --git a/integration/expo-native/src/core/instances/throwaway.ts b/integration/expo-native/src/core/instances/throwaway.ts new file mode 100644 index 00000000000..1a00b369a1c --- /dev/null +++ b/integration/expo-native/src/core/instances/throwaway.ts @@ -0,0 +1,503 @@ +import { randomBytes } from 'node:crypto'; +import { existsSync, mkdirSync, readFileSync, readdirSync, renameSync, rmSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { DEVELOPMENT_USER_LIMIT, REPLACE_AT_USERS, frontendApiHost, type ClerkBackend } from '../clerk.ts'; +import { sleep as defaultSleep } from '../exec.ts'; +import type { InstanceKeys } from '../keys.ts'; +import { count } from '../state.ts'; +import { newEntryId, type Workspace } from '../workspace.ts'; +import { VerifyFailure, type ApplicationView, type InstanceView, type Json, type LedgerEntry, type ProcessRef, type PublishableKey } from '../types.ts'; +import { compareEnvironment, describeDifference, type EnvironmentComparison, type Leaves } from './settings.ts'; +import { deadlineOf, throwawayApplication, throwawayName, type Listing, type OpenWorkspace, type Platform } from './platform.ts'; +import { STANDARD, STANDARD_ENVIRONMENT_KEY, STANDARD_FILE, SettingsRefused, configFor, configLeaves, expectedEnvironment, sameLeaf, settingsFileOf, settingsOf, type Settings, type SettingsGroup } from './settings.ts'; + +type ApplicationEntry = Extract; + +interface InstanceRecord { + readonly application: string; + readonly instanceId: string; + readonly publishableKey: PublishableKey; +} + +interface Held { + readonly entry: ApplicationEntry; + readonly record: InstanceRecord; +} + +interface ApplicationState { + readonly settings?: Settings; + readonly drivers: readonly ProcessRef[]; +} + +const DEFAULT_LIFETIME_HOURS = 6; +const LIFETIME_HOURS = { min: 2, max: 72 } as const; +const REPLACE_WITHIN_MS = 60 * 60_000; +const REAP_GRACE_MS = 5 * 60_000; +const REAP_PER_COMMAND = 5; +const SHOWN_WITHIN_MS = 15_000; +const MOVED_LEAVES_LISTED = 20; + +export interface ThrowawayDeps { + readonly workspace: Workspace; + readonly platform: Platform; + readonly clerk: (keys: () => InstanceKeys) => ClerkBackend; + readonly env: Readonly>; + readonly self: ProcessRef; + readonly isRunning: (driver: ProcessRef) => boolean; + readonly fetch?: typeof fetch; + readonly sleep?: (ms: number) => Promise; + readonly now?: () => number; +} + +export interface HeldApplication { + readonly id: string; + readonly name: string; + readonly settings: Settings | null; + keys(): Promise; +} + +export interface AppliedApplication { + readonly id: string; + readonly name: string; + readonly keys: InstanceKeys; + readonly changed: boolean; + stillApplied(): Promise; +} + +export interface Inspection { + readonly application: HeldApplication; + readonly found: EnvironmentComparison | 'gone'; +} + +export interface Throwaway { + held(): HeldApplication | null; + ensure(options: { readonly willChange: boolean }, progress: (line: string) => void): Promise<{ readonly views: readonly InstanceView[]; readonly created: HeldApplication | null }>; + apply(group: SettingsGroup, progress: (line: string) => void): Promise; + stopDriving(): void; + inspect(): Promise; + finish(ledger: Workspace, progress: (line: string) => void): Promise; +} + +export const openApplications = (ledger: Workspace): readonly ApplicationEntry[] => ledger.unclosedEntries().filter((entry): entry is ApplicationEntry => entry.kind === 'application'); + +const recordsDir = (ledger: Workspace): string => join(ledger.root, 'instances'); +const recordFile = (ledger: Workspace, name: string): string => join(recordsDir(ledger), `${name}.json`); +const stateFile = (ledger: Workspace, name: string): string => join(recordsDir(ledger), `${name}.state.json`); + +function writeWhole(file: string, text: string): void { + const staged = `${file}.${randomBytes(4).toString('hex')}.tmp`; + writeFileSync(staged, text, { mode: 0o600 }); + renameSync(staged, file); +} + +function readRecord(ledger: Workspace, name: string): InstanceRecord | null { + try { + const raw = JSON.parse(readFileSync(recordFile(ledger, name), 'utf8')) as Partial | null; + if (typeof raw?.application !== 'string' || typeof raw.instanceId !== 'string' || typeof raw.publishableKey !== 'string') return null; + return { application: raw.application, instanceId: raw.instanceId, publishableKey: raw.publishableKey }; + } catch { + return null; + } +} + +const UNKNOWN: ApplicationState = { drivers: [] }; + +function recordedSettings(raw: unknown): Settings | undefined { + const record = raw as Partial | null | undefined; + if (typeof record !== 'object' || record === null || typeof record.key !== 'string') return undefined; + try { + const fresh = settingsOf(record.declared ?? null, typeof record.askedBy === 'string' ? record.askedBy : null); + return fresh.key === record.key ? fresh : undefined; + } catch { + return undefined; + } +} + +function readState(ledger: Workspace, name: string): ApplicationState { + try { + const raw = JSON.parse(readFileSync(stateFile(ledger, name), 'utf8')) as { environmentKey?: unknown; settings?: unknown; drivers?: unknown } | null; + if (typeof raw !== 'object' || raw === null) return UNKNOWN; + const settings = raw.environmentKey === STANDARD_ENVIRONMENT_KEY ? recordedSettings(raw.settings) : undefined; + const drivers = Array.isArray(raw.drivers) ? (raw.drivers as Partial[]).flatMap((driver) => (typeof driver?.pid === 'number' && typeof driver.startedAt === 'number' ? [{ pid: driver.pid, startedAt: driver.startedAt }] : [])) : []; + return { ...(settings === undefined ? {} : { settings }), drivers }; + } catch { + return UNKNOWN; + } +} + +function writeState(ledger: Workspace, name: string, state: ApplicationState): void { + writeWhole(stateFile(ledger, name), `${JSON.stringify({ environmentKey: STANDARD_ENVIRONMENT_KEY, ...(state.settings === undefined ? {} : { settings: state.settings }), drivers: state.drivers })}\n`); +} + +function lifetimeMs(env: ThrowawayDeps['env']): number { + const raw = env.VERIFY_THROWAWAY_HOURS; + if (raw === undefined || raw === '') return DEFAULT_LIFETIME_HOURS * 3_600_000; + const hours = Number(raw); + if (!Number.isInteger(hours) || hours < LIFETIME_HOURS.min || hours > LIFETIME_HOURS.max) { + throw new VerifyFailure('USAGE', `VERIFY_THROWAWAY_HOURS=${raw} is not a whole number from ${LIFETIME_HOURS.min} to ${LIFETIME_HOURS.max}`, 'unset VERIFY_THROWAWAY_HOURS or set it within range'); + } + return hours * 3_600_000; +} + +const compareWith = (settings: Settings, live: Json): EnvironmentComparison => compareEnvironment(expectedEnvironment(settings), live); + +interface Moved { + readonly compared: number; + readonly answeredMs: number; + readonly visibleMs: number; +} + +const nearDeadline = (name: string, at: number): boolean => { + const deadline = deadlineOf(name); + return deadline !== null && deadline.getTime() - at < REPLACE_WITHIN_MS; +}; + +const seconds = (ms: number, digits: number): string => `${(ms / 1000).toFixed(digits)}s`; + +const fileOf = (settings: Settings): string => (settings.askedBy === null ? 'the settings file' : settingsFileOf(settings.askedBy)); + +const beyondDeclared = (compared: EnvironmentComparison, declared: Leaves): string | null => + compared.differing.some((d) => d.path in declared) ? null : JSON.stringify(compared.differing.map((d) => [d.path, d.found ?? null])); + +export function createThrowaway(deps: ThrowawayDeps): Throwaway { + const { workspace, platform, self } = deps; + const request = deps.fetch ?? fetch; + const sleep = deps.sleep ?? defaultSleep; + const now = deps.now ?? Date.now; + let listedOnce = false; + const secretKeys = new Map>(); + + const close = (ledger: Workspace, entry: ApplicationEntry): void => { + const dir = recordsDir(ledger); + const files = existsSync(dir) ? readdirSync(dir).filter((file) => file.startsWith(`${entry.name}.`)) : []; + for (const file of files) rmSync(join(dir, file), { force: true }); + ledger.append({ id: newEntryId(), kind: 'done', ref: entry.id }); + }; + + function pool(): { readonly held: Held | null; readonly strays: readonly ApplicationEntry[] } { + let held: Held | null = null; + const strays: ApplicationEntry[] = []; + for (const entry of openApplications(workspace)) { + const record = readRecord(workspace, entry.name); + if (record === null) strays.push(entry); + else held ??= { entry, record }; + } + return { held, strays }; + } + + async function keysOf(app: Held): Promise { + const id = app.record.application; + let sk = secretKeys.get(id); + if (sk === undefined) { + sk = platform.secretKey(throwawayApplication(id, app.entry.name)); + secretKeys.set(id, sk); + } + return { pk: app.record.publishableKey, sk: await sk }; + } + const describe = (app: Held, settings: Settings | undefined): HeldApplication => ({ id: app.record.application, name: app.entry.name, settings: settings ?? null, keys: () => keysOf(app) }); + const otherDrivers = (state: ApplicationState): readonly ProcessRef[] => state.drivers.filter((driver) => driver.pid !== self.pid && deps.isRunning(driver)); + + async function environment(pk: PublishableKey): Promise<{ readonly status: number; readonly json: Json; readonly at: number }> { + const response = await request(`https://${frontendApiHost(pk)}/v1/environment`, { signal: AbortSignal.timeout(15_000) }); + const text = await response.text(); + const date = Date.parse(response.headers.get('date') ?? ''); + let json: Json = null; + try { + json = JSON.parse(text) as Json; + } catch { + json = null; + } + return { status: response.status, json, at: Number.isNaN(date) ? now() : date }; + } + + async function listing(open: OpenWorkspace): Promise { + if (listedOnce) return open.list(); + listedOnce = true; + return open.opened; + } + + const unanswered = (app: Held, status: number): VerifyFailure => + new VerifyFailure('NOT_READY', `the Frontend API of ${app.record.application} (${app.entry.name}) answered ${status}`, 'rerun; a cloud environment needs *.clerk.accounts.dev in its allowed domains'); + + const standardRefused = (said: string): VerifyFailure => + new VerifyFailure('INSTANCE_MISCONFIGURED', `Clerk's Platform API refused the standard settings in ${STANDARD_FILE}: ${said}`, 'the standard file needs a change of its own; the spec is not at fault'); + + async function blame(open: OpenWorkspace, app: Held, to: Settings, refusal: SettingsRefused): Promise { + if (to.declared === null) return standardRefused(refusal.said); + try { + await open.configure({ application: throwawayApplication(app.record.application, app.entry.name), instanceId: app.record.instanceId }, configFor(STANDARD), { dryRun: true }); + } catch (error) { + if (error instanceof SettingsRefused) return standardRefused(error.said); + throw error; + } + const { param } = refusal; + const declared = Object.keys(configLeaves(to.declared.config)); + const what = + param === null + ? `Clerk refused these together; remove or correct one of ${declared.join(', ')} in \`config\` in ${fileOf(to)}.` + : declared.some((leaf) => leaf === param || leaf.endsWith(`.${param}`)) + ? `correct or remove \`${param}\` in \`config\` in ${fileOf(to)}: Clerk says what is wrong with it above.` + : `add \`${param}\` to \`config\` in ${fileOf(to)} (Clerk requires it with what the spec declares), or remove what requires it.`; + return new SettingsRefused( + `${to.askedBy} declares ${to.label}, and Clerk's Platform API refused it: ${refusal.said}`, + what, + refusal, + ); + } + + function notShown(app: Held, to: Settings, status: number, compared: EnvironmentComparison | null, afterMs: number): VerifyFailure { + const id = app.record.application; + if (compared === null) return new VerifyFailure('NOT_READY', `the Frontend API of ${id} (${app.entry.name}) answered ${status} after its settings were changed`, 'rerun; a cloud environment needs *.clerk.accounts.dev in its allowed domains'); + if (to.declared === null) { + return new VerifyFailure( + 'INSTANCE_MISCONFIGURED', + `${id} (${app.entry.name}) does not match ${STANDARD_FILE} after it was configured: ${compared.differing.slice(0, 5).map((d) => describeDifference(d)).join('; ')}`, + '`{cli} down`, then rerun once; if it repeats, Clerk changed what a setting does, and the file needs a change of its own, apart from the work being verified', + ); + } + const declared = to.declared.environment; + const unmet = compared.differing.filter((d) => d.path in declared); + const absent = unmet.filter((d) => d.found === undefined).map((d) => d.path); + const other = unmet.filter((d) => d.found !== undefined); + const moved = compared.differing.filter((d) => !(d.path in declared)); + const pairs = moved.slice(0, MOVED_LEAVES_LISTED).map((d) => `${JSON.stringify(d.path)}: ${d.found === undefined ? 'absent' : JSON.stringify(d.found)}`).join(', '); + const problems = [ + ...(absent.length === 0 ? [] : [{ said: `its public environment has no leaf ${absent.join(', ')}`, fix: `check the spelling of ${absent.join(', ')} under \`environment\` in ${fileOf(to)}` }]), + ...(other.length === 0 ? [] : [{ said: `it does not show ${other.slice(0, 5).map((d) => describeDifference(d, 'the declaration expects')).join('; ')}`, fix: `correct the leaves under \`environment\` in ${fileOf(to)} to what the setting shows in the instance's public environment` }]), + ...(moved.length === 0 ? [] : [{ said: `the change moved ${count(moved.length, 'setting')} the declaration does not list: ${pairs}${moved.length > MOVED_LEAVES_LISTED ? `, and ${moved.length - MOVED_LEAVES_LISTED} more` : ''}`, fix: `one setting can move several leaves: add them to \`environment\` in ${fileOf(to)}, as listed` }]), + ]; + return new SettingsRefused(`${to.askedBy} declares ${to.label}, and ${seconds(afterMs, 1)} after Clerk accepted it on ${id} ${problems.map((problem) => problem.said).join('; and ')}`, problems.map((problem) => problem.fix).join('; ')); + } + + async function moveTo(open: OpenWorkspace, app: Held, to: Settings): Promise { + const name = app.entry.name; + const before = readState(workspace, name); + writeState(workspace, name, { drivers: before.drivers }); + const body = configFor(to); + const started = now(); + let answer: { readonly after: Json }; + try { + answer = await open.configure({ application: throwawayApplication(app.record.application, name), instanceId: app.record.instanceId }, body); + } catch (error) { + if (!(error instanceof SettingsRefused)) throw error; + writeState(workspace, name, before); + throw await blame(open, app, to, error); + } + const answered = now(); + const after = configLeaves(answer.after ?? {}); + const differing = Object.entries(configLeaves(body)).filter(([leaf, value]) => !sameLeaf(after[leaf], value)); + const declared = to.declared === null ? {} : configLeaves(to.declared.config); + const altered = differing.find(([leaf]) => Object.hasOwn(declared, leaf) && after[leaf] !== undefined); + if (altered !== undefined) { + const [leaf, value] = altered; + throw new SettingsRefused(`${to.askedBy} declares ${leaf}=${JSON.stringify(value)}, and Clerk stored ${JSON.stringify(after[leaf])}`, `correct \`${leaf}\` in \`config\` in ${fileOf(to)} to a value Clerk keeps`); + } + const lost = differing[0]; + if (lost !== undefined) { + throw new VerifyFailure( + 'INSTANCE_MISCONFIGURED', + `Clerk accepted the settings for ${app.record.application} and its answer does not hold ${lost[0]}=${JSON.stringify(lost[1])} (it has ${after[lost[0]] === undefined ? 'no such key' : JSON.stringify(after[lost[0]])})`, + 'rerun once; if it repeats, the Platform API changed how it answers a config change, which is a verify bug to report', + ); + } + const until = answered + SHOWN_WITHIN_MS; + let settled: string | null = null; + for (;;) { + const live = await environment(app.record.publishableKey); + const compared = live.status !== 200 ? null : compareWith(to, live.json); + if (compared !== null && compared.differing.length === 0) { + writeState(workspace, name, { settings: to, drivers: before.drivers }); + return { compared: compared.compared, answeredMs: answered - started, visibleMs: now() - answered }; + } + const rest = compared === null || to.declared === null ? null : beyondDeclared(compared, to.declared.environment); + if ((rest !== null && rest === settled) || now() >= until) throw notShown(app, to, live.status, compared, now() - answered); + settled = rest; + await sleep(500); + } + } + + const changedLine = (settings: Settings, id: string, moved: Moved): string => + `settings ${settings.label} on ${id} in ${seconds(moved.answeredMs + moved.visibleMs, 1)} (Clerk answered in ${seconds(moved.answeredMs, 2)}, the instance showed it ${seconds(moved.visibleMs, 2)} later), ${moved.compared} settings match`; + + async function create(open: OpenWorkspace, at: Date | null, progress: (line: string) => void): Promise { + const name = throwawayName(new Date((at?.getTime() ?? now()) + lifetimeMs(deps.env)), randomBytes(4).toString('hex')); + const entry: ApplicationEntry = { id: newEntryId(), kind: 'application', name, workspace: open.workspace }; + workspace.append(entry); + progress(`instance creating ${name} in ${open.workspace}`); + const started = now(); + const created = await open.create(name); + const record: InstanceRecord = { application: created.application.id, instanceId: created.instanceId, publishableKey: created.pk }; + mkdirSync(recordsDir(workspace), { recursive: true, mode: 0o700 }); + writeFileSync(recordFile(workspace, name), `${JSON.stringify(record)}\n`, { mode: 0o600, flag: 'wx' }); + secretKeys.set(record.application, Promise.resolve(created.sk)); + const app: Held = { entry, record }; + const moved = await moveTo(open, app, STANDARD); + progress(`instance ${record.application} up in ${seconds(now() - started, 1)} on standard, ${moved.compared} settings match ${STANDARD_FILE}`); + return app; + } + + async function reap(open: OpenWorkspace, listed: Listing, progress: (line: string) => void): Promise { + const at = listed.at; + if (at === null) return; + const mine = new Set(openApplications(workspace).map((entry) => entry.name)); + const expired = listed.applications.filter((application) => { + const deadline = deadlineOf(application.name); + return !mine.has(application.name) && deadline !== null && deadline.getTime() + REAP_GRACE_MS < at.getTime(); + }); + for (const application of expired.slice(0, REAP_PER_COMMAND)) { + try { + await open.delete(application); + progress(`reap ${application.name} (its deadline passed and the session that made it never deleted it)`); + } catch (error) { + progress(`reap ${application.name} left in place: ${(error as Error).message}`); + } + } + if (expired.length > REAP_PER_COMMAND) progress(`reap ${expired.length - REAP_PER_COMMAND} more expired applications are left for the next command`); + } + + type Verdict = { readonly keep: 'as-is' | 'busy' } | { readonly repair: string } | { readonly retire: string }; + + async function judge(app: Held, state: ApplicationState): Promise { + if (otherDrivers(state).length > 0) return { keep: 'busy' }; + const live = await environment(app.record.publishableKey); + if (live.status === 404) return { retire: 'Clerk no longer serves it' }; + if (live.status !== 200) throw unanswered(app, live.status); + if (nearDeadline(app.entry.name, live.at)) return { retire: 'its deadline is near' }; + const keys = await keysOf(app); + const users = await deps.clerk(() => keys).userCount(); + if (users >= REPLACE_AT_USERS) return { retire: `it holds ${users} of the ${DEVELOPMENT_USER_LIMIT} users a development instance allows` }; + if (state.settings === undefined) return { repair: 'what it is on is not recorded' }; + if (compareWith(state.settings, live.json).differing.length > 0) return { repair: `it no longer shows ${state.settings.label}` }; + return { keep: 'as-is' }; + } + + return { + held: () => { + const app = pool().held; + return app === null ? null : describe(app, readState(workspace, app.entry.name).settings); + }, + + async inspect() { + const app = pool().held; + if (app === null) return null; + const state = readState(workspace, app.entry.name); + const live = await environment(app.record.publishableKey); + if (live.status !== 200 && live.status !== 404) throw new Error(`the Frontend API of ${app.record.application} answered ${live.status}`); + return { application: describe(app, state.settings), found: live.status === 404 ? 'gone' : compareWith(state.settings ?? STANDARD, live.json) }; + }, + + async ensure(options, progress) { + lifetimeMs(deps.env); + const { held, strays } = pool(); + const state = held === null ? UNKNOWN : readState(workspace, held.entry.name); + const verdict = held === null ? null : await judge(held, state); + const kept = held !== null && verdict !== null && !('retire' in verdict) ? { app: held, verdict } : null; + if (kept !== null && 'keep' in kept.verdict) { + progress(`instance ${kept.app.record.application} held, on ${state.settings?.label ?? 'unknown settings'}${kept.verdict.keep === 'busy' ? ', and another run in this worktree is driving on it' : ''}`); + } + const view = (app: Held, created: boolean): InstanceView => ({ id: app.record.application, name: app.entry.name, created, settings: readState(workspace, app.entry.name).settings?.label ?? 'unknown settings' }); + if (strays.length === 0 && kept !== null && 'keep' in kept.verdict && !options.willChange) return { views: [view(kept.app, false)], created: null }; + + const open = await platform.open(); + const listed = await listing(open); + const byName = new Map(listed.applications.map((application) => [application.name, application])); + const retired = new Set(); + const retire = async (entry: ApplicationEntry, id: string | null, why: string): Promise => { + const application = byName.get(entry.name) ?? (id === null ? undefined : throwawayApplication(id, entry.name)); + if (application !== undefined) await open.delete(application); + close(workspace, entry); + retired.add(entry.name); + progress(`instance ${id ?? entry.name} retired (${why})`); + }; + for (const entry of strays) await retire(entry, null, 'its keys are lost'); + if (held !== null && verdict !== null && 'retire' in verdict) await retire(held.entry, held.record.application, verdict.retire); + + let failure: unknown; + if (kept !== null && 'repair' in kept.verdict) { + progress(`settings changing ${kept.app.record.application} from ${state.settings?.label ?? 'unknown settings'} to standard, because ${kept.verdict.repair}`); + await moveTo(open, kept.app, STANDARD).then((moved) => progress(changedLine(STANDARD, kept.app.record.application, moved)), (error: unknown) => (failure = error)); + } + let created: Held | null = null; + if (kept === null) created = await create(open, listed.at, progress).catch((error: unknown) => ((failure = error), null)); + + await reap(open, { ...listed, applications: listed.applications.filter((application) => !retired.has(application.name)) }, progress); + if (failure !== undefined) throw failure; + return { views: [...(kept === null ? [] : [view(kept.app, false)]), ...(created === null ? [] : [view(created, true)])], created: created === null ? null : describe(created, STANDARD) }; + }, + + async apply(group, progress) { + const to = group.settings; + const app = pool().held; + if (app === null) throw new VerifyFailure('NOT_READY', 'this worktree holds no application to put on the settings of a group', '{cli} up'); + const id = app.record.application; + const name = app.entry.name; + const state = readState(workspace, name); + const other = otherDrivers(state)[0]; + if (other !== undefined) { + throw new VerifyFailure('DEVICE_BUSY', `another run in this worktree (pid ${other.pid}) is driving on ${id}, and a worktree has one application, which serves one run at a time`, 'let that run finish, then rerun'); + } + const keys = await keysOf(app); + let changed = false; + const live = state.settings?.key === to.key ? await environment(app.record.publishableKey) : null; + if (live !== null && live.status === 200 && compareWith(to, live.json).differing.length === 0) { + progress(`settings ${to.label} already on ${id}`); + } else { + progress(`settings changing ${id} from ${state.settings?.label ?? 'unknown settings'} to ${to.declared === null ? `standard, for ${group.specs[0]?.path ?? 'this run'}` : `${to.label}, which ${to.askedBy} declares`}`); + progress(changedLine(to, id, await moveTo(await platform.open(), app, to))); + changed = true; + } + writeState(workspace, name, { settings: to, drivers: [self] }); + return { + id, + name, + keys, + changed, + async stillApplied() { + const shown = await environment(app.record.publishableKey); + if (shown.status !== 200) throw unanswered(app, shown.status); + return compareWith(to, shown.json).differing.length === 0; + }, + }; + }, + + stopDriving() { + const app = pool().held; + if (app === null) return; + const current = readState(workspace, app.entry.name); + if (current.drivers.some((driver) => driver.pid === self.pid)) writeState(workspace, app.entry.name, { ...current, drivers: otherDrivers(current) }); + }, + + async finish(ledger, progress) { + const owned = openApplications(ledger); + if (owned.length === 0) return []; + const open = await platform.open(); + const foreign = owned.filter((entry) => entry.workspace !== open.workspace); + const mine = owned.filter((entry) => entry.workspace === open.workspace); + const listed = await listing(open); + const byName = new Map(listed.applications.map((application) => [application.name, application])); + const targets = mine.flatMap((entry) => { + const id = readRecord(ledger, entry.name)?.application; + return byName.get(entry.name) ?? (id === undefined ? [] : [throwawayApplication(id, entry.name)]); + }); + const deletions = await Promise.allSettled(targets.map((application) => open.delete(application))); + const after = await open.list(); + const remaining = new Set(after.applications.map((application) => application.name)); + const gone = mine.filter((entry) => !remaining.has(entry.name)); + for (const entry of gone) close(ledger, entry); + await reap(open, after, progress); + + const left = [...mine.filter((entry) => remaining.has(entry.name)), ...foreign]; + if (left.length > 0) { + const refusal = deletions.find((result) => result.status === 'rejected'); + const why = foreign.length > 0 ? `${foreign.map((entry) => entry.name).join(', ')} belong to workspace ${foreign[0]!.workspace}, which this credential does not reach` : refusal === undefined ? 'Clerk still lists them' : (refusal.reason as Error).message; + const one = foreign.length === 1; + const fixes = [ + ...(left.length > foreign.length ? ['{cli} down again; it deletes what is left'] : []), + ...(foreign.length === 0 ? [] : [`${foreign.map((entry) => entry.name).join(', ')} ${one ? 'belongs' : 'belong'} to workspace ${foreign[0]!.workspace}, which this credential cannot reach; only a credential of that workspace can delete ${one ? 'it' : 'them'}`]), + ]; + throw new VerifyFailure('NOT_READY', `${count(left.length, 'application')} of this worktree could not be deleted: ${why}`, fixes.join('. ')); + } + return gone.map((entry) => ({ name: entry.name })); + }, + }; +} From 907885afe5aaffce086a21513e6b4036060944af Mon Sep 17 00:00:00 2001 From: Mike Pitre <12040919+mikepitre@users.noreply.github.com> Date: Wed, 7 Oct 2026 20:30:59 -0400 Subject: [PATCH 4/7] test(expo): add the platform host and device backends Co-Authored-By: Claude Opus 5.5 --- integration/expo-native/specs/app.ts | 39 ++ integration/expo-native/src/fixture.ts | 231 ++++++++ integration/expo-native/src/freshness.ts | 370 ++++++++++++ integration/expo-native/src/host.ts | 533 ++++++++++++++++++ .../src/platform/android/emulator.ts | 45 ++ .../expo-native/src/platform/android/local.ts | 452 +++++++++++++++ .../expo-native/src/platform/android/sdk.ts | 170 ++++++ .../expo-native/src/platform/ios/local.ts | 286 ++++++++++ .../expo-native/src/platform/ios/simulator.ts | 19 + 9 files changed, 2145 insertions(+) create mode 100644 integration/expo-native/specs/app.ts create mode 100644 integration/expo-native/src/fixture.ts create mode 100644 integration/expo-native/src/freshness.ts create mode 100644 integration/expo-native/src/host.ts create mode 100644 integration/expo-native/src/platform/android/emulator.ts create mode 100644 integration/expo-native/src/platform/android/local.ts create mode 100644 integration/expo-native/src/platform/android/sdk.ts create mode 100644 integration/expo-native/src/platform/ios/local.ts create mode 100644 integration/expo-native/src/platform/ios/simulator.ts diff --git a/integration/expo-native/specs/app.ts b/integration/expo-native/specs/app.ts new file mode 100644 index 00000000000..570c569cb8d --- /dev/null +++ b/integration/expo-native/specs/app.ts @@ -0,0 +1,39 @@ +import type { TestApp } from './support/inputs.ts'; +import type { AppEntry, Platform } from './support/types.ts'; + +export const APP_ID = 'com.clerk.exponativebuildfixture'; + +const ANDROID_ACTIVITY = '.MainActivity'; +const DEV_CLIENT_SCHEME = 'exp+clerk-expo-native-build-fixture'; + +export function devClientEntry(platform: Platform, devServer: string): AppEntry { + return platform === 'ios' + ? { + kind: 'dev-client', + launchArguments: [ + '--initialUrl', + devServer, + '-EXDevMenuShowsAtLaunch', + 'NO', + '-EXDevMenuIsOnboardingFinished', + 'YES', + '-EXDevMenuShowFloatingActionButton', + 'NO', + ], + openLink: null, + androidActivity: null, + } + : { + kind: 'dev-client', + launchArguments: [], + openLink: `${DEV_CLIENT_SCHEME}://expo-development-client/?url=${encodeURIComponent(devServer)}`, + androidActivity: ANDROID_ACTIVITY, + }; +} + +export const app: TestApp = { + platforms: ['ios', 'android'], + id: (): string => APP_ID, + entry: (platform: Platform, devServer: string | null): AppEntry => + devServer === null ? { kind: 'binary' } : devClientEntry(platform, devServer), +}; diff --git a/integration/expo-native/src/fixture.ts b/integration/expo-native/src/fixture.ts new file mode 100644 index 00000000000..91bcf309112 --- /dev/null +++ b/integration/expo-native/src/fixture.ts @@ -0,0 +1,231 @@ +import { spawn } from 'node:child_process'; +import { cpSync, existsSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { VerifyFailure, type Platform } from './core/types.ts'; +import { resolveJavaHome, sdkRoot } from './platform/android/sdk.ts'; + +export const IOS_PRODUCT = 'ClerkExpoNativeBuildFixture'; +export const WORKTREE = fileURLToPath(new URL('../../../', import.meta.url)); +export const FIXTURE = join(WORKTREE, 'integration', 'templates', 'expo-native'); + +export type BuildProduct = 'dev-client'; + +interface Recipe { + readonly configuration: 'Debug' | 'Release'; + readonly gradle: readonly string[]; + readonly apk: string; + readonly expoPackages: readonly string[]; + readonly label: string; +} + +const EXPO_PACKAGES = ['expo-auth-session', 'expo-constants', 'expo-crypto', 'expo-secure-store', 'expo-web-browser']; + +const RECIPES: Readonly> = { + 'dev-client': { + configuration: 'Debug', + gradle: ['assembleDebug'], + apk: join('debug', 'app-debug.apk'), + expoPackages: [...EXPO_PACKAGES, 'expo-dev-client'], + label: 'dev client', + }, +}; + +const SHARED_NATIVE_INPUTS = [ + 'packages/expo/app.plugin.js', + 'packages/expo/src/specs', + 'packages/expo/expo-module.config.json', + 'packages/expo/react-native.config.js', + 'packages/expo/package.json', + 'packages/expo-google-signin/app.plugin.js', + 'packages/expo-google-signin/expo-module.config.json', + 'packages/expo-google-signin/package.json', + 'packages/expo-biometrics/expo-module.config.json', + 'packages/expo-biometrics/package.json', + 'integration/templates/expo-native/app.json', + 'integration/templates/expo-native/app.config.js', + 'integration/templates/expo-native/package.sdk-57.json', + 'integration/templates/expo-native/pnpm-workspace.yaml', + 'integration/templates/expo-native/modules', +] as const; + +const PLATFORM_NATIVE_INPUTS: Readonly> = { + ios: ['packages/expo/ios', 'packages/expo-google-signin/ios', 'packages/expo-biometrics/ios'], + android: ['packages/expo/android', 'packages/expo-google-signin/android', 'packages/expo-biometrics/android'], +}; + +export function nativeInputs(platform: Platform): readonly string[] { + return [...PLATFORM_NATIVE_INPUTS[platform], ...SHARED_NATIVE_INPUTS]; +} + +export function artifact(platform: Platform, product: BuildProduct, fixture: string = FIXTURE): string { + const recipe = RECIPES[product]; + return platform === 'ios' + ? join( + fixture, + 'ios', + 'build', + 'Build', + 'Products', + `${recipe.configuration}-iphonesimulator`, + `${IOS_PRODUCT}.app`, + ) + : join(fixture, 'android', 'app', 'build', 'outputs', 'apk', recipe.apk); +} + +function step( + command: string, + args: readonly string[], + cwd: string, + env: Readonly> = {}, +): Promise<{ code: number; tail: string }> { + return new Promise(resolve => { + const child = spawn(command, [...args], { + cwd, + env: { ...process.env, LANG: 'en_US.UTF-8', CI: '1', ...env }, + stdio: ['ignore', 'pipe', 'pipe'], + }); + const lines: string[] = []; + const collect = (chunk: Buffer) => { + for (const line of chunk.toString().split('\n')) { + if (line.trim().length === 0) continue; + lines.push(line); + if (lines.length > 40) lines.shift(); + } + }; + child.stdout.on('data', collect); + child.stderr.on('data', collect); + child.on('error', () => resolve({ code: 127, tail: `${command} could not start` })); + child.on('close', code => resolve({ code: code ?? 1, tail: lines.slice(-15).join('\n') })); + }); +} + +export async function mustStep( + what: string, + command: string, + args: readonly string[], + cwd: string, + env?: Readonly>, +): Promise { + const result = await step(command, args, cwd, env); + if (result.code !== 0) + throw new VerifyFailure( + 'BUILD_FAILED', + `${what} exited ${result.code}:\n${result.tail}`, + 'fix the error above, then rerun {cli} up', + ); +} + +export interface FixtureBuild { + readonly platform: Platform; + readonly product: BuildProduct; + readonly nativeKey: string; + readonly buildPackages: boolean; + readonly progress: (line: string) => void; +} + +export interface FixtureSite { + readonly worktree: string; + readonly fixture: string; + readonly must: typeof mustStep; +} + +const THIS_CHECKOUT: FixtureSite = { worktree: WORKTREE, fixture: FIXTURE, must: mustStep }; + +const nativeProjectMarker = (site: FixtureSite, platform: Platform) => + join(site.fixture, platform, '.verify-native-project'); + +export function nativeProjectIsCurrent(site: FixtureSite, platform: Platform, wanted: string): boolean { + const marker = nativeProjectMarker(site, platform); + return ( + existsSync(join(site.fixture, 'node_modules')) && existsSync(marker) && readFileSync(marker, 'utf8') === wanted + ); +} + +async function generateNativeProject(site: FixtureSite, build: FixtureBuild, wanted: string): Promise { + const { platform, progress } = build; + rmSync(nativeProjectMarker(site, platform), { force: true }); + cpSync(join(site.fixture, 'package.sdk-57.json'), join(site.fixture, 'package.json')); + await site.must( + 'pnpm add the workspace packages', + 'pnpm', + [ + 'add', + 'link:../../../packages/expo', + 'link:../../../packages/expo-google-signin', + 'link:../../../packages/expo-biometrics', + ], + site.fixture, + ); + await site.must('expo install', 'pnpm', ['expo', 'install', ...RECIPES[build.product].expoPackages], site.fixture); + progress(`build expo prebuild --clean --platform ${platform}`); + await site.must('expo prebuild', 'pnpm', ['expo', 'prebuild', '--clean', '--platform', platform], site.fixture); + writeFileSync(nativeProjectMarker(site, platform), wanted); +} + +export async function buildFixture(build: FixtureBuild, site: FixtureSite = THIS_CHECKOUT): Promise { + const { platform, product, progress } = build; + const recipe = RECIPES[product]; + if (!existsSync(join(site.worktree, 'node_modules'))) { + throw new VerifyFailure('NOT_READY', 'the monorepo has no node_modules', `cd ${site.worktree} && pnpm install`); + } + if (build.buildPackages) { + progress('build turbo build @clerk/expo, @clerk/expo-biometrics, @clerk/expo-google-signin'); + await site.must( + 'turbo build', + 'pnpm', + [ + 'turbo', + 'build', + '--filter=@clerk/expo...', + '--filter=@clerk/expo-biometrics...', + '--filter=@clerk/expo-google-signin...', + ], + site.worktree, + ); + } + const wanted = `${product} ${build.nativeKey}`; + if (nativeProjectIsCurrent(site, platform, wanted)) { + progress(`build the ${platform} project was generated from these native inputs, so expo prebuild is skipped`); + } else { + await generateNativeProject(site, build, wanted); + } + if (platform === 'ios') { + progress(`build xcodebuild ${recipe.configuration} (${recipe.label})`); + await site.must( + 'xcodebuild', + 'xcodebuild', + [ + 'build', + '-quiet', + '-workspace', + `ios/${IOS_PRODUCT}.xcworkspace`, + '-scheme', + IOS_PRODUCT, + '-configuration', + recipe.configuration, + '-sdk', + 'iphonesimulator', + '-derivedDataPath', + join(site.fixture, 'ios', 'build'), + 'CODE_SIGN_IDENTITY=-', + ], + site.fixture, + ); + } else { + const java = resolveJavaHome(); + if (!java.ok) throw new VerifyFailure('NOT_READY', java.detail, java.fix); + progress(`build gradlew ${recipe.gradle.at(-1)} (${recipe.label})`); + await site.must( + `gradlew ${recipe.gradle.at(-1)}`, + './gradlew', + [...recipe.gradle, '-q'], + join(site.fixture, 'android'), + { + JAVA_HOME: java.home, + ANDROID_HOME: sdkRoot(), + }, + ); + } + return artifact(platform, product, site.fixture); +} diff --git a/integration/expo-native/src/freshness.ts b/integration/expo-native/src/freshness.ts new file mode 100644 index 00000000000..7830309c3e6 --- /dev/null +++ b/integration/expo-native/src/freshness.ts @@ -0,0 +1,370 @@ +import { createHash } from 'node:crypto'; +import { existsSync, readFileSync, readdirSync, realpathSync, statSync } from 'node:fs'; +import { join, relative } from 'node:path'; + +interface SourceFile { + readonly rel: string; + readonly mtime: number; +} + +const BUILT_EXTENSION = /\.(ts|tsx|js|jsx)$/; + +const inTestsDir = (rel: string) => rel.split('/').includes('__tests__'); + +export function isTsdownSource(rel: string): boolean { + return BUILT_EXTENSION.test(rel) && !/\.test\.(ts|tsx)$/.test(rel) && !inTestsDir(rel); +} + +export function isPackageSource(rel: string): boolean { + return ( + BUILT_EXTENSION.test(rel) && + !/\.(test|spec)\.(ts|tsx|js|jsx)$/.test(rel) && + !inTestsDir(rel) && + !rel.endsWith('.d.ts') + ); +} + +export function isBundledOutput(rel: string): boolean { + return /\.(js|cjs|mjs)$/.test(rel); +} + +export function listFiles(root: string, prefix: string, keep: (rel: string) => boolean): readonly SourceFile[] { + const walk = (dir: string, relDir: string): SourceFile[] => { + if (!existsSync(dir)) return []; + return readdirSync(dir, { withFileTypes: true }).flatMap(entry => { + const rel = relDir === '' ? entry.name : `${relDir}/${entry.name}`; + const path = join(dir, entry.name); + if (entry.isDirectory()) return entry.name === 'node_modules' ? [] : walk(path, rel); + return entry.isFile() && keep(rel) ? [{ rel: `${prefix}${rel}`, mtime: statSync(path).mtimeMs }] : []; + }); + }; + return walk(root, ''); +} + +export const newest = (files: readonly SourceFile[]): number => files.reduce((max, f) => Math.max(max, f.mtime), 0); + +interface WorkspacePackage { + readonly name: string; + readonly dir: string; +} + +export function workspaceDependencies(worktree: string, packageDir: string): readonly WorkspacePackage[] { + const packagesRoot = realpathSync(join(worktree, 'packages')); + const found = new Map(); + const visit = (dir: string) => { + const manifest = JSON.parse(readFileSync(join(dir, 'package.json'), 'utf8')) as Record< + string, + Record | undefined + >; + const names = Object.keys({ + ...manifest.dependencies, + ...manifest.peerDependencies, + ...manifest.optionalDependencies, + }); + for (const name of names) { + const link = join(dir, 'node_modules', name); + if (found.has(name) || !existsSync(link)) continue; + const real = realpathSync(link); + if (relative(packagesRoot, real).startsWith('..') || !existsSync(join(real, 'package.json'))) continue; + found.set(name, { name, dir: real }); + visit(real); + } + }; + visit(packageDir); + return [...found.values()]; +} + +function isExpoModule(pkg: WorkspacePackage): boolean { + return existsSync(join(pkg.dir, 'expo-module.config.json')); +} + +const builtTo = (pkg: WorkspacePackage) => listFiles(join(pkg.dir, 'dist'), '', isBundledOutput); +const hasBuild = (pkg: WorkspacePackage) => existsSync(join(pkg.dir, 'src')) && existsSync(join(pkg.dir, 'dist')); + +interface BuiltRecord { + readonly sources: string; + readonly dist: string; +} +export type BuiltRecords = Readonly>; + +function contentDigest(files: readonly SourceFile[]): string { + const hash = createHash('sha256'); + for (const file of [...files].sort((a, b) => a.rel.localeCompare(b.rel))) { + hash.update(`${file.rel}\0`).update(readFileSync(file.rel)).update('\0'); + } + return hash.digest('hex'); +} + +function outputSignature(files: readonly SourceFile[]): string { + return [...files] + .sort((a, b) => a.rel.localeCompare(b.rel)) + .map(f => `${f.rel}:${f.mtime}`) + .join('|'); +} + +interface Judged { + readonly stale: boolean; + readonly record: BuiltRecord | null; +} + +function judgeBuilt( + pkg: WorkspacePackage, + sourcePackages: readonly WorkspacePackage[], + previous: BuiltRecord | undefined, +): Judged { + const sources = sourcePackages.flatMap(p => listFiles(join(p.dir, 'src'), `${p.dir}/src/`, isPackageSource)); + const dist = builtTo(pkg); + const signature = outputSignature(dist); + if (!isStale(sources, dist)) return { stale: false, record: { sources: contentDigest(sources), dist: signature } }; + if (previous === undefined || previous.dist !== signature) return { stale: true, record: null }; + const unchanged = previous.sources === contentDigest(sources); + return { stale: !unchanged, record: unchanged ? previous : null }; +} + +interface ScopeCheck { + readonly stale: readonly WorkspacePackage[]; + readonly records: BuiltRecords; +} + +function check(worktree: string, expoDir: string, records: BuiltRecords, inScope: boolean): ScopeCheck { + const next: Record = { ...records }; + const stale: WorkspacePackage[] = []; + for (const pkg of workspaceDependencies(worktree, expoDir)) { + if (isExpoModule(pkg) !== inScope || !hasBuild(pkg)) continue; + const sourcePackages = inScope ? [pkg] : [pkg, ...workspaceDependencies(worktree, pkg.dir).filter(hasBuild)]; + const judged = judgeBuilt(pkg, sourcePackages, records[pkg.name]); + if (judged.stale) { + stale.push(pkg); + } else if (judged.record !== null) { + next[pkg.name] = judged.record; + } + } + return { stale, records: next }; +} + +export function staleInScope(worktree: string, expoDir: string, records: BuiltRecords = {}): ScopeCheck { + return check(worktree, expoDir, records, true); +} + +export function staleOutOfScope(worktree: string, expoDir: string, records: BuiltRecords = {}): ScopeCheck { + return check(worktree, expoDir, records, false); +} + +export function isStale(srcFiles: readonly SourceFile[], distFiles: readonly SourceFile[]): boolean { + return srcFiles.length > 0 && distFiles.length > 0 && newest(srcFiles) > newest(distFiles); +} + +interface OutputEntry { + readonly mtime: number; + readonly size: number; + readonly hash: string; + readonly since: number; +} +export type Fingerprint = Readonly>; + +export function fingerprint(root: string, rels: readonly string[], previous: Fingerprint | null): Fingerprint { + const out: Record = {}; + for (const rel of rels) { + const path = join(root, rel); + let stat; + try { + stat = statSync(path); + } catch { + continue; + } + const before = previous?.[rel]; + if (before !== undefined && before.mtime === stat.mtimeMs && before.size === stat.size) { + out[rel] = before; + continue; + } + let content: Buffer; + try { + content = readFileSync(path); + } catch { + continue; + } + const hash = createHash('sha256').update(content).digest('hex'); + out[rel] = { + mtime: stat.mtimeMs, + size: stat.size, + hash, + since: before !== undefined && before.hash === hash ? before.since : stat.mtimeMs, + }; + } + return out; +} + +export function changedFiles(before: Fingerprint, after: Fingerprint): readonly string[] { + const rels = new Set([...Object.keys(before), ...Object.keys(after)]); + return [...rels].filter(rel => before[rel]?.hash !== after[rel]?.hash).sort(); +} + +const sameContent = (a: Fingerprint, b: Fingerprint): boolean => changedFiles(a, b).length === 0; + +function inBundle(body: string, rels: readonly string[]): readonly string[] { + return rels.filter(rel => body.includes(`${rel}"`)); +} + +function bundledDigest(current: Fingerprint, body: string): string { + const hash = createHash('sha256'); + for (const rel of inBundle(body, Object.keys(current)).slice().sort()) hash.update(`${rel}:${current[rel]!.hash}\n`); + return hash.digest('hex'); +} + +const STALE = 'stale'; + +export interface GateMemory { + readonly metroPid: number; + readonly spawn: Fingerprint | null; + readonly seen: Readonly>; + readonly outputs: Fingerprint; +} + +export interface BundleView { + readonly revId: string; + readonly lastModified: number; + readonly body: string; +} + +type Verdict = 'fresh' | 'stale' | 'restart'; + +export function judge( + memory: GateMemory, + current: Fingerprint, + bundle: BundleView, +): { readonly verdict: Verdict; readonly memory: GateMemory } { + const digest = bundledDigest(current, bundle.body); + const known = memory.seen[bundle.revId]; + if (known !== undefined) return { verdict: known === digest ? 'fresh' : 'stale', memory }; + let verdict: Verdict; + if (Object.keys(memory.seen).length === 0) { + verdict = memory.spawn !== null && bundledDigest(memory.spawn, bundle.body) === digest ? 'fresh' : 'restart'; + } else { + const lastContent = Math.max(0, ...inBundle(bundle.body, Object.keys(current)).map(rel => current[rel]!.since)); + verdict = bundle.lastModified >= Math.floor(lastContent / 1000) * 1000 ? 'fresh' : 'stale'; + } + if (verdict === 'restart') return { verdict, memory }; + return { + verdict, + memory: { ...memory, seen: { ...memory.seen, [bundle.revId]: verdict === 'fresh' ? digest : STALE } }, + }; +} + +export type Fetched = + | { readonly ok: true; readonly value: T } + | { readonly ok: false; readonly transient: boolean; readonly message: string }; + +export interface GateIO { + list(): readonly string[]; + fingerprint(rels: readonly string[], previous: Fingerprint | null): Fingerprint; + fetch(): Promise>; + touch(rels: readonly string[]): void; + restart(): Promise; + now(): number; + sleep(ms: number): Promise; + progress(line: string): void; +} + +interface GateOptions { + readonly timeoutMs: number; + readonly nudgeAfterMs: number; + readonly maxRestarts: number; + readonly settledFailureMs: number; +} + +type GateResult = + | { readonly ok: true; readonly memory: GateMemory } + | { readonly ok: false; readonly kind: 'timeout' | 'bundle-error'; readonly message: string }; + +const complete = (rels: readonly string[], fp: Fingerprint) => rels.length === Object.keys(fp).length; + +export async function confirmServed(io: GateIO, start: GateMemory, options: GateOptions): Promise { + let memory = start; + let current = memory.outputs; + let candidate: { readonly revId: string; readonly outputs: Fingerprint } | null = null; + let failure: { + readonly message: string; + readonly outputs: Fingerprint; + readonly rels: string; + readonly since: number; + } | null = null; + let lastError = ''; + let staleSince: number | null = null; + let restarts = 0; + let delay = 500; + const deadline = io.now() + options.timeoutMs; + while (io.now() < deadline) { + const rels = io.list(); + const before = io.fingerprint(rels, current); + const bundle = await io.fetch(); + const after = io.fingerprint(io.list(), before); + current = after; + const settled = sameContent(before, after) && complete(rels, after) && complete(io.list(), after); + if (!bundle.ok) { + candidate = null; + lastError = bundle.message; + const key = rels.join('\n'); + if (!bundle.transient && settled) { + if ( + failure !== null && + failure.message === bundle.message && + failure.rels === key && + sameContent(failure.outputs, after) + ) { + if (io.now() - failure.since >= options.settledFailureMs) + return { ok: false, kind: 'bundle-error', message: bundle.message }; + } else { + failure = { message: bundle.message, outputs: after, rels: key, since: io.now() }; + } + } else { + failure = null; + } + await io.sleep(delay); + delay = Math.min(delay * 2, 8_000); + continue; + } + failure = null; + delay = 500; + if (!settled) { + candidate = null; + await io.sleep(500); + continue; + } + const judged = judge(memory, after, bundle.value); + memory = judged.memory; + if (judged.verdict === 'restart') { + if (restarts >= options.maxRestarts) + return { ok: false, kind: 'timeout', message: 'Metro kept serving a bundle that predates the current dist' }; + restarts += 1; + io.progress( + 'metro restarting Metro, because dist changed after it started and its first bundle cannot be dated', + ); + const spawn = io.fingerprint(io.list(), after); + const pid = await io.restart(); + memory = { metroPid: pid, spawn, seen: {}, outputs: memory.outputs }; + candidate = null; + continue; + } + if (judged.verdict === 'stale') { + candidate = null; + staleSince ??= io.now(); + if (io.now() - staleSince >= options.nudgeAfterMs) { + const missed = inBundle(bundle.value.body, changedFiles(memory.outputs, after)); + io.progress( + `metro still serving an older bundle; touching ${missed.length} changed file(s) so Metro's watcher sees them`, + ); + io.touch(missed); + staleSince = io.now(); + } + await io.sleep(500); + continue; + } + staleSince = null; + if (candidate !== null && candidate.revId === bundle.value.revId && sameContent(candidate.outputs, after)) { + return { ok: true, memory: { ...memory, outputs: after } }; + } + candidate = { revId: bundle.value.revId, outputs: after }; + await io.sleep(500); + } + return { ok: false, kind: 'timeout', message: lastError }; +} diff --git a/integration/expo-native/src/host.ts b/integration/expo-native/src/host.ts new file mode 100644 index 00000000000..4a8bb643425 --- /dev/null +++ b/integration/expo-native/src/host.ts @@ -0,0 +1,533 @@ +import { spawn } from 'node:child_process'; +import { + appendFileSync, + cpSync, + existsSync, + mkdirSync, + openSync, + readFileSync, + rmSync, + utimesSync, + writeFileSync, +} from 'node:fs'; +import { createRequire } from 'node:module'; +import { join, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { isRunning, run, sleep } from './core/exec.ts'; +import { + LOCAL_POOL, + VerifyFailure, + type HostAdapter, + type Platform, + type RuntimeProcess, + type ScratchPath, +} from './core/types.ts'; +import { takeSlotLock } from './core/workspace.ts'; +import { FIXTURE, IOS_PRODUCT, WORKTREE, buildFixture, mustStep, nativeInputs } from './fixture.ts'; +import { APP_ID, app } from '../specs/app.ts'; +import { + confirmServed, + fingerprint, + isBundledOutput, + isTsdownSource, + listFiles, + newest, + staleInScope, + staleOutOfScope, + workspaceDependencies, + type BuiltRecords, + type BundleView, + type Fetched, + type Fingerprint, + type GateMemory, +} from './freshness.ts'; +import { localAndroidBackend } from './platform/android/local.ts'; +import { sdkTool } from './platform/android/sdk.ts'; +import { localIosBackend } from './platform/ios/local.ts'; + +const ANDROID_DEV_MENU_PREFS = ``; + +const GITHUB_REPO = 'clerk/javascript'; +const EXPO_PACKAGE = join(WORKTREE, 'packages', 'expo'); +const RUNTIME_DIR = fileURLToPath(new URL('../.verify/runtime/', import.meta.url)); + +const FIXTURE_LOCK_MINUTES = 45; + +export function metroPort(lease: { readonly platform: Platform; readonly slot: number }): number { + return 8081 + (lease.platform === 'ios' ? 0 : 4) + lease.slot; +} + +async function withFixtureLock(progress: (line: string) => void, fn: () => Promise): Promise { + mkdirSync(RUNTIME_DIR, { recursive: true }); + const release = await takeSlotLock( + join(RUNTIME_DIR, 'fixture-lock'), + FIXTURE_LOCK_MINUTES * 60_000, + () => + new VerifyFailure( + 'NOT_READY', + `another verify process in this worktree has been building the expo-native fixture for ${FIXTURE_LOCK_MINUTES} minutes`, + '{cli} down, then {cli} up', + ), + owner => + progress(`build waiting for pid ${owner.pid}, which is building the expo-native fixture in this worktree`), + ); + try { + return await fn(); + } finally { + release(); + } +} + +const pidFile = (name: string) => join(RUNTIME_DIR, `${name}.json`); +const runtimeLog = (name: string) => join(RUNTIME_DIR, `${name}.log`); + +function readRuntime(name: string): RuntimeProcess | null { + if (!existsSync(pidFile(name))) return null; + const ref = JSON.parse(readFileSync(pidFile(name), 'utf8')) as RuntimeProcess; + return isRunning(ref) ? ref : null; +} + +const allRuntimeNames = (): readonly string[] => [ + 'watch', + ...(['ios', 'android'] as const).flatMap(platform => + Array.from({ length: LOCAL_POOL[platform] }, (_, i) => `metro-${metroPort({ platform, slot: i + 1 })}`), + ), +]; + +function stopRuntime(names: readonly string[] = allRuntimeNames()): void { + for (const name of names) { + const ref = readRuntime(name); + if (ref !== null) process.kill(ref.pid, 'SIGTERM'); + rmSync(pidFile(name), { force: true }); + } +} + +function startDetached( + name: string, + owner: Pick, + args: readonly string[], + cwd: string, +): RuntimeProcess { + mkdirSync(RUNTIME_DIR, { recursive: true }); + const log = openSync(runtimeLog(name), 'a'); + appendFileSync(log, `\n==== ${name} started ${new Date().toISOString()}\n`); + const { CI: _ci, ...env } = process.env; + const child = spawn(process.execPath, [...args], { + cwd, + detached: true, + stdio: ['ignore', log, log], + env: { ...env, LANG: 'en_US.UTF-8', EXPO_NO_TELEMETRY: '1', EXPO_OFFLINE: '1' }, + }); + child.unref(); + const ref: RuntimeProcess = { ...owner, pid: child.pid ?? 0, startedAt: Date.now() }; + writeFileSync(pidFile(name), JSON.stringify(ref)); + return ref; +} + +async function waitFor( + what: string, + ready: () => Promise, + timeoutMs: number, + logName: string, + fix?: string, +): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (await ready()) return; + await sleep(500); + } + throw new VerifyFailure( + 'NOT_READY', + `${what} was not ready within ${timeoutMs / 1000}s`, + fix ?? `read ${runtimeLog(logName)}`, + ); +} + +interface Started { + readonly names: string[]; +} + +export async function withCleanup( + stop: (names: readonly string[]) => void, + fn: (started: Started) => Promise, + progress: (line: string) => void, +): Promise { + const started: Started = { names: [] }; + try { + return await fn(started); + } catch (error) { + if (started.names.length > 0) { + progress(`stop ${started.names.join(', ')} (started by this call, which failed)`); + stop(started.names); + } + throw error; + } +} + +async function ensureWatch(started: Started, progress: (line: string) => void): Promise { + const running = readRuntime('watch'); + if (running !== null) return running; + const tsdown = createRequire(join(EXPO_PACKAGE, 'package.json')).resolve('tsdown/package.json'); + const offset = existsSync(runtimeLog('watch')) ? readFileSync(runtimeLog('watch')).length : 0; + const ref = startDetached( + 'watch', + { what: 'watch' }, + [join(tsdown, '..', 'dist', 'run.mjs'), '--watch'], + EXPO_PACKAGE, + ); + started.names.push('watch'); + progress(`watch packages/expo tsdown --watch (pid ${ref.pid})`); + await waitFor( + 'the @clerk/expo watch build', + async () => /Build complete|built in|Rebuilt/i.test(readFileSync(runtimeLog('watch')).subarray(offset).toString()), + 180_000, + 'watch', + ); + return ref; +} + +const expoOutputs = () => listFiles(join(EXPO_PACKAGE, 'dist'), 'packages/expo/dist/', isBundledOutput); + +async function waitForWatchToCatchUp(): Promise { + let previous = ''; + await waitFor( + 'the @clerk/expo watch build to pick up the latest edit', + async () => { + const outputs = expoOutputs(); + const signature = outputs.map(f => `${f.rel}:${f.mtime}`).join('|'); + const settled = signature === previous; + previous = signature; + return ( + settled && newest(outputs) >= newest(listFiles(join(EXPO_PACKAGE, 'src'), 'packages/expo/src/', isTsdownSource)) + ); + }, + 60_000, + 'watch', + ); +} + +const recordsFile = () => join(RUNTIME_DIR, 'built-dependencies.json'); + +function readRecords(): BuiltRecords { + return existsSync(recordsFile()) ? (JSON.parse(readFileSync(recordsFile(), 'utf8')) as BuiltRecords) : {}; +} + +function writeRecords(records: BuiltRecords): void { + mkdirSync(RUNTIME_DIR, { recursive: true }); + writeFileSync(recordsFile(), JSON.stringify(records)); +} + +function refuseOutOfScope(): void { + const { stale, records } = staleOutOfScope(WORKTREE, EXPO_PACKAGE, readRecords()); + writeRecords(records); + if (stale.length === 0) return; + const names = stale.map(pkg => pkg.name); + throw new VerifyFailure( + 'NOT_READY', + `${names.join(', ')} ${stale.length === 1 ? 'has' : 'have'} source newer than ${stale.length === 1 ? 'its dist' : 'their dist'}, with content that dist was not built from. This CLI rebuilds only @clerk/expo and its Expo-module siblings; it verifies other workspace dependencies after you build them, and it will not launch on their stale dist`, + `{cli} down, then pnpm turbo build --force ${names.map(n => `--filter=${n}`).join(' ')}, then rerun`, + ); +} + +async function rebuildStaleSiblings(progress: (line: string) => void): Promise { + const { stale, records } = staleInScope(WORKTREE, EXPO_PACKAGE, readRecords()); + writeRecords(records); + if (stale.length === 0) return; + const names = stale.map(pkg => pkg.name); + progress( + `build ${names.join(', ')} src changed since dist was built; stopping this worktree's Metro and watch build, then pnpm turbo build --force ${names.map(n => `--filter=${n}`).join(' ')}`, + ); + stopRuntime(); + await mustStep( + `turbo build ${names.join(' ')}`, + 'pnpm', + ['turbo', 'build', '--force', ...names.map(n => `--filter=${n}`)], + WORKTREE, + ); + writeRecords(staleInScope(WORKTREE, EXPO_PACKAGE, readRecords()).records); +} + +function servedOutputs(): readonly string[] { + const roots = [{ name: '@clerk/expo', dir: EXPO_PACKAGE }, ...workspaceDependencies(WORKTREE, EXPO_PACKAGE)]; + return roots.flatMap(pkg => { + const prefix = `${relative(WORKTREE, pkg.dir)}/dist/`; + return listFiles(join(pkg.dir, 'dist'), prefix, isBundledOutput).map(f => f.rel); + }); +} + +async function metroAnswers(port: number): Promise { + try { + const response = await fetch(`http://127.0.0.1:${port}/status`, { signal: AbortSignal.timeout(2000) }); + return (await response.text()).includes('packager-status:running'); + } catch { + return false; + } +} + +interface Metro { + readonly ref: RuntimeProcess; + readonly spawnOutputs: Fingerprint | null; + readonly started: Started; +} + +const metroCli = () => join(FIXTURE, 'node_modules', 'expo', 'bin', 'cli'); + +const startMetro = (port: number, platform: Platform): RuntimeProcess => + startDetached( + `metro-${port}`, + { what: 'metro', platform }, + [metroCli(), 'start', '--port', String(port), '--dev-client'], + FIXTURE, + ); + +async function ensureMetro( + started: Started, + port: number, + platform: Platform, + progress: (line: string) => void, +): Promise { + const name = `metro-${port}`; + const running = readRuntime(name); + if (running !== null) { + await waitFor( + `Metro pid ${running.pid} on port ${port} to answer`, + () => metroAnswers(port), + 120_000, + name, + `{cli} down, then retry; read ${runtimeLog(name)}`, + ); + return { ref: running, spawnOutputs: null, started }; + } + if (await metroAnswers(port)) { + throw new VerifyFailure( + 'NOT_READY', + `port ${port} already serves a Metro that this worktree did not start`, + `stop the process listening on ${port} (lsof -nP -iTCP:${port} -sTCP:LISTEN)`, + ); + } + if (!existsSync(metroCli())) + throw new VerifyFailure('NOT_READY', 'the expo-native fixture has no node_modules', '{cli} up'); + const spawnOutputs = fingerprint(WORKTREE, servedOutputs(), null); + const ref = startMetro(port, platform); + started.names.push(name); + progress(`metro :${port} expo start (pid ${ref.pid})`); + await waitFor(`Metro on port ${port}`, () => metroAnswers(port), 120_000, name); + return { ref, spawnOutputs, started }; +} + +async function fetchManifest(port: number, platform: Platform): Promise> { + try { + const response = await fetch(`http://127.0.0.1:${port}/`, { + headers: { 'expo-platform': platform, accept: 'application/expo+json,application/json' }, + signal: AbortSignal.timeout(30_000), + }); + const text = await response.text(); + const url = response.ok ? (JSON.parse(text) as { launchAsset?: { url?: string } }).launchAsset?.url : undefined; + return url === undefined + ? { ok: false, transient: true, message: `manifest ${response.status}: ${text.slice(0, 200)}` } + : { ok: true, value: url }; + } catch (error) { + return { ok: false, transient: true, message: (error as Error).message }; + } +} + +async function fetchBundle(url: string): Promise> { + try { + const response = await fetch(url, { signal: AbortSignal.timeout(300_000) }); + const body = await response.text(); + if (response.status === 200) + return { + ok: true, + value: { + revId: response.headers.get('x-metro-delta-id') ?? '', + lastModified: Date.parse(response.headers.get('last-modified') ?? '') || 0, + body, + }, + }; + let message = body.slice(0, 300); + try { + const parsed = JSON.parse(body) as { type?: string; message?: string }; + message = `${parsed.type ?? 'error'}: ${(parsed.message ?? '').split('\n')[0]}`; + } catch {} + return { ok: false, transient: response.status !== 500, message: `bundle ${response.status}: ${message}` }; + } catch (error) { + return { ok: false, transient: true, message: (error as Error).message }; + } +} + +async function ensureServed( + metro: Metro, + port: number, + platform: Platform, + progress: (line: string) => void, +): Promise { + const name = `metro-${port}`; + const stateFile = join(RUNTIME_DIR, `served-${port}-${platform}.json`); + const saved = existsSync(stateFile) ? (JSON.parse(readFileSync(stateFile, 'utf8')) as Partial) : null; + let current = metro.ref; + const memory: GateMemory = + saved !== null && saved.metroPid === current.pid && saved.seen !== undefined && saved.outputs !== undefined + ? (saved as GateMemory) + : { metroPid: current.pid, spawn: metro.spawnOutputs, seen: {}, outputs: metro.spawnOutputs ?? {} }; + const pending = + Object.keys(memory.seen).length === 0 + ? [] + : servedOutputs().filter( + rel => fingerprint(WORKTREE, [rel], memory.outputs)[rel]?.hash !== memory.outputs[rel]?.hash, + ); + progress( + Object.keys(memory.seen).length === 0 + ? `metro :${port} bundling ${platform} once so the first launch does not wait on Metro` + : pending.length > 0 + ? `metro :${port} ${pending.length} served file(s) changed since the last launch; waiting until Metro's ${platform} bundle has the new code` + : `metro :${port} no served file changed since the last launch`, + ); + const fix = `retry {cli} up --platform ${platform}; if it fails again, read ${runtimeLog(name)}`; + const url = await (async () => { + let last = ''; + let delay = 500; + const deadline = Date.now() + 120_000; + while (Date.now() < deadline) { + const manifest = await fetchManifest(port, platform); + if (manifest.ok) return manifest.value; + last = manifest.message; + await sleep(delay); + delay = Math.min(delay * 2, 8_000); + } + throw new VerifyFailure( + 'NOT_READY', + `Metro on port ${port} returned no ${platform} launch asset within 120s (${last})`, + fix, + ); + })(); + const result = await confirmServed( + { + list: servedOutputs, + fingerprint: (rels, previous) => fingerprint(WORKTREE, rels, previous), + fetch: () => fetchBundle(url), + touch: rels => { + const now = new Date(); + for (const rel of rels) utimesSync(join(WORKTREE, rel), now, now); + }, + restart: async () => { + if (isRunning(current)) process.kill(current.pid, 'SIGTERM'); + await waitFor(`Metro pid ${current.pid} to exit`, async () => !(await metroAnswers(port)), 30_000, name); + current = startMetro(port, platform); + metro.started.names.includes(name) || metro.started.names.push(name); + progress(`metro :${port} expo start (pid ${current.pid})`); + await waitFor(`Metro on port ${port}`, () => metroAnswers(port), 120_000, name); + return current.pid; + }, + now: Date.now, + sleep, + progress: line => progress(line.replace(/^metro {3}/, `metro :${port} `)), + }, + memory, + { timeoutMs: 300_000, nudgeAfterMs: 5_000, maxRestarts: 2, settledFailureMs: 3_000 }, + ); + if (!result.ok) { + if (result.kind === 'bundle-error') { + throw new VerifyFailure( + 'NOT_READY', + `Metro on port ${port} could not bundle ${platform}: ${result.message}`, + `fix the bundling error, then rerun; read ${runtimeLog(name)}`, + ); + } + throw new VerifyFailure( + 'NOT_READY', + `Metro on port ${port} did not serve the latest ${platform} bundle within 300s${result.message === '' ? '' : ` (${result.message})`}`, + fix, + ); + } + writeFileSync(stateFile, JSON.stringify(result.memory)); + return current; +} + +function keepBuild(platform: Platform, built: string, into: string): string { + mkdirSync(into, { recursive: true }); + const path = join(into, platform === 'ios' ? `${IOS_PRODUCT}.app` : 'app-debug.apk'); + rmSync(path, { recursive: true, force: true }); + cpSync(built, path, { recursive: true, verbatimSymlinks: true }); + return path; +} + +export const host: HostAdapter = { + repo: 'clerk-expo', + cli: 'integration/expo-native/bin/control-clerk-expo', + platforms: ['ios', 'android'], + githubRepo: GITHUB_REPO, + appId: app.id, + buildInputs: platform => nativeInputs(platform), + async build(platform, key, into, progress) { + if (process.platform !== 'darwin') + throw new VerifyFailure( + 'UNSUPPORTED', + `the expo-native fixture is built locally on macOS only, and this machine runs ${process.platform}`, + 'run this command on a Mac', + ); + const path = await withFixtureLock(progress, async () => { + const watching = readRuntime('watch') !== null; + if (watching) + progress('build the running watch build keeps packages/expo/dist current, so turbo build is skipped'); + else stopRuntime(); + const built = await buildFixture({ + platform, + product: 'dev-client', + nativeKey: key, + buildPackages: !watching, + progress, + }); + return keepBuild(platform, built, into); + }); + return { platform, key, appId: APP_ID, path: path as ScratchPath, source: 'local' }; + }, + async runtime(lease, progress) { + const port = metroPort(lease); + return withCleanup( + stopRuntime, + async started => { + refuseOutOfScope(); + await rebuildStaleSiblings(progress); + const watch = await ensureWatch(started, progress); + await waitForWatchToCatchUp(); + const metro = await ensureServed( + await ensureMetro(started, port, lease.platform, progress), + port, + lease.platform, + progress, + ); + if (lease.platform === 'android') { + const adb = sdkTool('adb'); + const reverse = await run(adb, ['-s', lease.deviceId, 'reverse', `tcp:${port}`, `tcp:${port}`]); + if (reverse.code !== 0) + throw new VerifyFailure( + 'NOT_READY', + `adb reverse tcp:${port} failed: ${reverse.stderr.trim()}`, + '{cli} down --platform android, then {cli} up --platform android', + ); + await run(adb, ['-s', lease.deviceId, 'shell', 'am', 'force-stop', APP_ID]); + const prefs = await run( + adb, + [ + '-s', + lease.deviceId, + 'shell', + `run-as ${APP_ID} sh -c 'mkdir -p shared_prefs && cat > shared_prefs/expo.modules.devmenu.sharedpreferences.xml'`, + ], + { input: ANDROID_DEV_MENU_PREFS }, + ); + if (prefs.code !== 0) + throw new VerifyFailure( + 'NOT_READY', + `could not turn off the dev menu onboarding: ${prefs.stderr.trim()}`, + '{cli} down --platform android, then {cli} up --platform android', + ); + } + return { devServer: `http://localhost:${port}`, processes: [watch, metro] }; + }, + progress, + ); + }, + logPredicates: { ios: `process == "${IOS_PRODUCT}" AND senderImagePath CONTAINS "${IOS_PRODUCT}"` }, + backends: [localIosBackend(), localAndroidBackend()], +}; diff --git a/integration/expo-native/src/platform/android/emulator.ts b/integration/expo-native/src/platform/android/emulator.ts new file mode 100644 index 00000000000..b16ae051173 --- /dev/null +++ b/integration/expo-native/src/platform/android/emulator.ts @@ -0,0 +1,45 @@ +import { AVD_NAME } from './sdk.ts'; + +export const RECORD_SIZE = '720x1608'; +export const LOG_FILTER = ['ClerkVerify:V', 'ClerkLog:V', 'OkHttp:V', 'ReactNativeJS:V', 'AndroidRuntime:E', '*:S']; + +const SOFTWARE_GPU = ['-gpu', 'swiftshader_indirect']; +const WITHOUT_THE_180_SECOND_CAP = ['--time-limit', '0']; +const SHELL_WRITABLE_BEFORE_STORAGE_MOUNTS = '/data/local/tmp'; + +export function emulatorArgs(port: number, os: NodeJS.Platform): readonly string[] { + return ['-avd', AVD_NAME, '-read-only', '-no-window', '-no-audio', '-no-boot-anim', ...(os === 'linux' ? SOFTWARE_GPU : []), '-port', String(port)]; +} + +export const LANE_SETTINGS = [ + ['window_animation_scale', '0'], + ['transition_animation_scale', '0'], + ['animator_duration_scale', '0'], + ['hide_error_dialogs', '1'], +] as const; + +export const laneSettingsCommand = (): string => LANE_SETTINGS.map(([name, value]) => `settings put global ${name} ${value}`).join(' && '); + +export const laneSettingsReadCommand = (): string => LANE_SETTINGS.map(([name]) => `settings get global ${name}`).join(' && '); + +export function laneSettingsHold(read: string): boolean { + const values = read.split('\n').map((line) => line.trim()).filter((line) => line.length > 0); + return values.length === LANE_SETTINGS.length && LANE_SETTINGS.every(([, wanted], index) => Number(values[index]) === Number(wanted)); +} + +export const installArgs = (apk: string): readonly string[] => ['install', '-r', '-t', apk]; + +export function logFilter(extraPredicate?: string | null): readonly string[] { + const extra = extraPredicate?.split(/\s+/).filter((spec) => spec.length > 0) ?? []; + return [...LOG_FILTER.slice(0, -1), ...extra, '*:S']; +} + +export function logcatSince(since: Date): string { + return (since.getTime() / 1000).toFixed(3); +} + +export const logcatArgs = (since: Date, extraPredicate?: string | null): readonly string[] => ['logcat', '-d', '-v', 'threadtime', '-T', logcatSince(since), ...logFilter(extraPredicate)]; + +export const recordingOnDevice = (name: string): string => `${SHELL_WRITABLE_BEFORE_STORAGE_MOUNTS}/verify-${name}.mp4`; + +export const screenrecordArgs = (deviceFile: string): readonly string[] => ['shell', 'screenrecord', '--size', RECORD_SIZE, ...WITHOUT_THE_180_SECOND_CAP, deviceFile]; diff --git a/integration/expo-native/src/platform/android/local.ts b/integration/expo-native/src/platform/android/local.ts new file mode 100644 index 00000000000..e934171bae7 --- /dev/null +++ b/integration/expo-native/src/platform/android/local.ts @@ -0,0 +1,452 @@ +import { execFileSync, spawn } from 'node:child_process'; +import { closeSync, existsSync, mkdirSync, openSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { basename, join } from 'node:path'; +import { defaultClaimsDir, freeSlot, isOrphaned, readClaim, readClaims, takeSlot, type Claim } from '../../core/claims.ts'; +import { isRunning, run, sleep, type ProcessRef } from '../../core/exec.ts'; +import { + LOCAL_POOL, + VerifyFailure, + type AcquireRequest, + type DeviceBackend, + type DoctorCheck, + type EvidencePath, + type LocalLease, + type Recording, +} from '../../core/types.ts'; +import { LANE_SETTINGS, emulatorArgs, installArgs, laneSettingsCommand, laneSettingsHold, laneSettingsReadCommand, logcatArgs, recordingOnDevice, screenrecordArgs } from './emulator.ts'; +import { AVD_NAME, ensureLaneAvd, jdkCheck, localAvailability, sdkTool, thisMachine, type Machine } from './sdk.ts'; + +const LOCALE = 'en-US'; +const BOOT_TIMEOUT_MS = 240_000; +const STILL_WAITING_MS = 60_000; +const LANE_PROPERTY = 'debug.verify.lane'; + +export const lanePort = (slot: number): number => 5558 + 2 * slot; +export const laneSerial = (slot: number): string => `emulator-${lanePort(slot)}`; + +export function parseAdbDevices(stdout: string): ReadonlyMap { + return new Map( + stdout + .split('\n') + .slice(1) + .map((line) => line.trim().split(/\s+/)) + .filter((parts): parts is [string, string, ...string[]] => parts.length >= 2 && parts[0] !== '') + .map(([serial, state]) => [serial, state]), + ); +} + +const listsAvd = (stdout: string) => stdout.split('\n').some((line) => line.trim() === AVD_NAME); + +export interface LocalAndroidOptions { + readonly claimsDir?: string; + readonly adbBin?: string; + readonly emulatorBin?: string; + readonly emulatorsDir?: string; + readonly machine?: Machine; +} + +interface SpawnedEmulator extends ProcessRef { + readonly nonce: string; +} + +function commandOf(pid: number): string { + try { + return execFileSync('ps', ['-o', 'command=', '-p', String(pid)], { encoding: 'utf8' }).trim(); + } catch { + return ''; + } +} + +export function terminateGroup(pid: number): void { + try { + process.kill(-pid, 'SIGTERM'); + } catch (error) { + const { code } = error as NodeJS.ErrnoException; + if (code !== 'ESRCH' && code !== 'EPERM') throw error; + } +} + +const hasArgument = (command: string, flag: string, value: string) => new RegExp(`(^|\\s)${flag} ${value}(\\s|$)`).test(command); + +function retryFix(request: AcquireRequest): string { + return request.retryWith.replace('', String(Math.max(600, request.waitSeconds * 2))); +} + +export function localAndroidBackend(options: LocalAndroidOptions = {}): DeviceBackend { + const claimsDir = options.claimsDir ?? defaultClaimsDir(); + const machine = options.machine ?? thisMachine(); + const emulatorLogDir = options.emulatorsDir ?? join(machine.home, '.verify', 'emulators'); + const adbBin = options.adbBin ?? sdkTool('adb', machine); + const emulatorBin = options.emulatorBin ?? sdkTool('emulator', machine); + + const adb = (serial: string, args: readonly string[]) => run(adbBin, ['-s', serial, ...args]); + const shell = async (serial: string, command: string) => (await adb(serial, ['shell', command])).stdout.trim(); + + async function devices(): Promise> { + const listed = await run(adbBin, ['devices']); + if (listed.code !== 0) throw new VerifyFailure('NOT_READY', `adb devices failed: ${listed.stderr.trim()}`, 'install the Android SDK platform-tools, or set ANDROID_HOME'); + return parseAdbDevices(listed.stdout); + } + + async function avdName(serial: string): Promise { + const result = await adb(serial, ['emu', 'avd', 'name']); + return result.code === 0 ? (result.stdout.split('\n')[0]?.trim() ?? null) : null; + } + + async function isOwnLane(serial: string, nonce: string): Promise { + return (await avdName(serial)) === AVD_NAME && (await shell(serial, `getprop ${LANE_PROPERTY}`)) === nonce; + } + + async function describeForeign(serial: string): Promise { + return `${serial} (${(await avdName(serial)) ?? 'unknown AVD'}, not a verify lane)`; + } + + const pidFile = (slot: number) => join(emulatorLogDir, `android-${slot}.pid`); + + function ownedProcess(slot: number, nonce: string): SpawnedEmulator | null { + let recorded: SpawnedEmulator; + try { + recorded = JSON.parse(readFileSync(pidFile(slot), 'utf8')) as SpawnedEmulator; + } catch { + return null; + } + if (recorded.nonce !== nonce || !isRunning(recorded)) return null; + const command = commandOf(recorded.pid); + return hasArgument(command, '-avd', AVD_NAME) && hasArgument(command, '-port', String(lanePort(slot))) ? recorded : null; + } + + function forgetProcess(slot: number, nonce: string): void { + try { + if ((JSON.parse(readFileSync(pidFile(slot), 'utf8')) as SpawnedEmulator).nonce === nonce) rmSync(pidFile(slot), { force: true }); + } catch { + return; + } + } + + async function killEmulator(slot: number, nonce: string): Promise { + const serial = laneSerial(slot); + const marked = (await devices()).has(serial) && (await isOwnLane(serial, nonce)); + const owned = ownedProcess(slot, nonce); + if (marked) await adb(serial, ['emu', 'kill']); + else if (owned !== null) process.kill(-owned.pid, 'SIGTERM'); + else { + forgetProcess(slot, nonce); + return; + } + const deadline = Date.now() + 30_000; + let forcedAt: number | null = null; + while ((await devices()).has(serial) || (owned !== null && isRunning(owned))) { + if (forcedAt !== null && Date.now() >= forcedAt + 10_000 && owned !== null && !isRunning(owned)) break; + if (forcedAt === null && Date.now() >= deadline) { + if (owned === null) throw new VerifyFailure('NOT_READY', `${serial} did not exit after \`adb emu kill\``, `adb -s ${serial} emu kill, then rerun the verb`); + try { + process.kill(-owned.pid, 'SIGKILL'); + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ESRCH') throw error; + } + forcedAt = Date.now(); + } + await sleep(1000); + } + forgetProcess(slot, nonce); + } + + async function waitForBoot(serial: string, exited: () => string | null): Promise { + const deadline = Date.now() + BOOT_TIMEOUT_MS; + for (;;) { + const failure = exited(); + if (failure !== null) throw new VerifyFailure('NOT_READY', `the ${AVD_NAME} emulator exited before it booted: ${failure}`, 'run `{cli} doctor`, then `{cli} up` again'); + if ((await shell(serial, 'getprop sys.boot_completed')) === '1' && (await shell(serial, 'getprop init.svc.bootanim')) !== 'running') return; + if (Date.now() >= deadline) throw new VerifyFailure('NOT_READY', `${serial} did not finish booting in ${BOOT_TIMEOUT_MS / 1000}s`, '{cli} down, then {cli} up'); + await sleep(2000); + } + } + + async function pinLocale(serial: string, progress: (line: string) => void): Promise { + const locale = (await shell(serial, 'getprop persist.sys.locale')) || (await shell(serial, 'getprop ro.product.locale')); + if (locale === LOCALE) return; + progress(`device ${serial} setting locale ${LOCALE}`); + await shell(serial, `setprop persist.sys.locale ${LOCALE}; setprop ctl.restart zygote`); + await sleep(3000); + const deadline = Date.now() + 120_000; + while ((await shell(serial, 'getprop init.svc.zygote')) !== 'running' || !(await shell(serial, 'pm path android')).startsWith('package:')) { + if (Date.now() >= deadline) throw new VerifyFailure('NOT_READY', `${serial} did not come back after the locale change`, '{cli} down, then {cli} up'); + await sleep(2000); + } + } + + async function applyLaneSettings(serial: string): Promise { + await shell(serial, laneSettingsCommand()); + const read = await shell(serial, laneSettingsReadCommand()); + if (!laneSettingsHold(read)) { + const names = LANE_SETTINGS.map(([name]) => name).join(', '); + throw new VerifyFailure('NOT_READY', `${serial} did not take the lane settings: ${names} read back as ${read.split('\n').join(', ') || 'nothing'}`, '{cli} down, then {cli} up'); + } + } + + async function lanePortsCheck(): Promise { + const running = await devices(); + const live = readClaims(claimsDir, 'android').filter((c) => !isOrphaned(c)); + const foreign: string[] = []; + for (let slot = 1; slot <= LOCAL_POOL.android; slot += 1) { + const serial = laneSerial(slot); + if (!running.has(serial)) continue; + const claim = live.find((c) => c.slot === slot); + const booting = claim !== undefined && ownedProcess(slot, claim.nonce) !== null; + if (claim === undefined || (!booting && !(await isOwnLane(serial, claim.nonce)))) foreign.push(serial); + } + if (foreign.length === 0) return { id: 'lane-ports', ok: true, detail: `ports ${lanePort(1)} to ${lanePort(LOCAL_POOL.android)} hold only verify lanes` }; + return { + id: 'lane-ports', + ok: false, + detail: `${(await Promise.all(foreign.map(describeForeign))).join(', ')} sits on a lane port and takes a lane from every worktree`, + fix: `${foreign.map((serial) => `adb -s ${serial} emu kill`).join('; ')}, but only if that emulator is yours; verify never kills it`, + }; + } + + async function claimSlot(request: AcquireRequest): Promise { + const startedWaiting = Date.now(); + const deadline = startedWaiting + request.waitSeconds * 1000; + let lastWait = ''; + let lastPrinted = 0; + for (;;) { + const running = await devices(); + const live = readClaims(claimsDir, 'android').filter((c) => !isOrphaned(c)); + const foreign = Array.from({ length: LOCAL_POOL.android }, (_, i) => i + 1).filter( + (slot) => running.has(laneSerial(slot)) && !live.some((c) => c.slot === slot), + ); + const inUse = [ + ...live.map((c) => `${c.deviceName} (held by ${c.worktree})`), + ...(await Promise.all(foreign.map((slot) => describeForeign(laneSerial(slot))))), + ].sort(); + const foreignFix = foreign.length === 0 ? '' : `; ${foreign.map((slot) => `\`adb -s ${laneSerial(slot)} emu kill\``).join(' or ')} frees a lane, but only if that emulator is yours`; + if (inUse.length < LOCAL_POOL.android) { + for (let slot = 1; slot <= LOCAL_POOL.android; slot += 1) { + if (foreign.includes(slot)) continue; + const { gen, claim: holder } = readClaim(claimsDir, 'android', slot); + if (holder !== null && !isOrphaned(holder)) continue; + const claim = takeSlot(claimsDir, 'android', slot, gen, request.worktree); + if (claim !== null) return claim; + } + } + if (Date.now() >= deadline) { + throw new VerifyFailure( + 'POOL_FULL', + `${inUse.length} of ${LOCAL_POOL.android} Android lanes are in use on this machine (${inUse.join(', ')})`, + `rerun with a longer --wait than ${request.waitSeconds}s, for example ${retryFix(request)}, or run {cli} down in a worktree that no longer needs its lane${foreignFix}`, + ); + } + const waiting = `wait all ${LOCAL_POOL.android} Android lanes are in use (${inUse.join(', ')}); waiting up to ${request.waitSeconds}s for one to free`; + if (waiting !== lastWait || Date.now() - lastPrinted >= STILL_WAITING_MS) { + request.progress(waiting === lastWait ? `wait still waiting after ${Math.round((Date.now() - startedWaiting) / 1000)}s (${inUse.join(', ')})` : waiting); + lastPrinted = Date.now(); + } + lastWait = waiting; + await sleep(5000); + } + } + + async function clearSlot(claim: Claim, worktree: string): Promise { + const reaper = takeSlot(claimsDir, 'android', claim.slot, claim.gen, worktree, true); + if (reaper === null) return; + await killEmulator(claim.slot, claim.nonce); + freeSlot(claimsDir, reaper); + } + + function bootEmulator(slot: number, nonce: string): { readonly pid: number | undefined; readonly exited: () => string | null; readonly stop: () => void } { + mkdirSync(emulatorLogDir, { recursive: true }); + const logFile = join(emulatorLogDir, `android-${slot}.log`); + const out = openSync(logFile, 'w'); + const child = spawn(emulatorBin, [...emulatorArgs(lanePort(slot), machine.os)], { detached: true, stdio: ['ignore', out, out] }); + closeSync(out); + if (child.pid !== undefined) { + const spawned: SpawnedEmulator = { nonce, pid: child.pid, startedAt: Date.now() }; + writeFileSync(pidFile(slot), JSON.stringify(spawned)); + } + let exit: string | null = null; + child.on('error', (error) => (exit = error.message)); + child.on('exit', (code) => { + const tail = existsSync(logFile) ? readFileSync(logFile, 'utf8').trim().split('\n').slice(-5).join(' | ') : ''; + exit = `exit ${code}${tail ? `: ${tail}` : ''}`; + }); + child.unref(); + return { + pid: child.pid, + exited: () => exit, + stop: () => { + if (exit === null && child.pid !== undefined) terminateGroup(child.pid); + }, + }; + } + + const backend: DeviceBackend = { + kind: 'local', + platform: 'android', + availability: () => localAvailability(machine), + requirement: 'a machine with the Android SDK emulator, adb, and an Android 36 Google APIs system image, and on Linux a /dev/kvm this user can open', + + async acquire(request) { + if (ensureLaneAvd(machine) === 'created') request.progress(`device wrote the ${AVD_NAME} AVD, which this machine did not have`); + const listed = await run(emulatorBin, ['-list-avds']); + if (!listsAvd(listed.stdout)) { + throw new VerifyFailure('NOT_READY', `the emulator does not list ${AVD_NAME}: ${(listed.stderr || listed.stdout).trim() || `exit ${listed.code}`}`, 'run `{cli} doctor`; if ANDROID_AVD_HOME is set, the AVD must be under it'); + } + const claim = await claimSlot(request); + const serial = laneSerial(claim.slot); + let boot: ReturnType | null = null; + const interrupted = (signal: NodeJS.Signals) => { + if (boot !== null) { + boot.stop(); + request.progress(`device boot cancelled; stopped emulator ${boot.pid} on ${serial}`); + } + process.removeListener('SIGINT', interrupted); + process.removeListener('SIGTERM', interrupted); + process.kill(process.pid, signal); + }; + process.once('SIGINT', interrupted); + process.once('SIGTERM', interrupted); + try { + if ((await devices()).has(serial)) { + throw new VerifyFailure('POOL_FULL', `${await describeForeign(serial)} took the lane port while it was being claimed`, retryFix(request)); + } + request.progress(`device ${claim.deviceName} booting ${AVD_NAME} -read-only on port ${lanePort(claim.slot)}`); + boot = bootEmulator(claim.slot, claim.nonce); + await waitForBoot(serial, boot.exited); + const died = boot.exited(); + if (died !== null) { + throw new VerifyFailure('NOT_READY', `the ${AVD_NAME} emulator verify started exited (${died}), so ${serial} is someone else's`, retryFix(request)); + } + await shell(serial, `setprop ${LANE_PROPERTY} ${claim.nonce}`); + if (!(await isOwnLane(serial, claim.nonce))) { + const marker = (await shell(serial, 'getprop')).split('\n').filter((line) => line.includes('verify.lane')).join(' ') || 'no verify.lane property'; + throw new VerifyFailure('NOT_READY', `${serial} is not the ${AVD_NAME} lane this claim booted (${marker})`, retryFix(request)); + } + await pinLocale(serial, request.progress); + await applyLaneSettings(serial); + } catch (error) { + boot?.stop(); + await clearSlot(claim, request.worktree).catch(() => undefined); + throw error; + } finally { + process.removeListener('SIGINT', interrupted); + process.removeListener('SIGTERM', interrupted); + } + return { + backend: 'local', + platform: 'android', + slot: claim.slot, + deviceName: claim.deviceName, + deviceId: serial, + claimNonce: claim.nonce, + acquiredAt: new Date().toISOString(), + installedBuild: null, + }; + }, + + async check(lease) { + if (readClaim(claimsDir, 'android', lease.slot).claim?.nonce !== lease.claimNonce) return 'lost'; + if ((await devices()).get(lease.deviceId) !== 'device') return 'lost'; + if (!(await isOwnLane(lease.deviceId, lease.claimNonce))) return 'lost'; + return (await shell(lease.deviceId, 'getprop sys.boot_completed')) === '1' ? 'held' : 'lost'; + }, + + async install(lease, app) { + if (app.path === null) throw new VerifyFailure('BUILD_FAILED', `build ${app.key} was made on another machine, so there is no app here to install on ${lease.deviceName}`, '{cli} down, then {cli} up'); + const result = await adb(lease.deviceId, installArgs(app.path)); + if (result.code !== 0 || !/Success/.test(result.stdout)) { + throw new VerifyFailure('NOT_READY', `adb install on ${lease.deviceName} failed: ${(result.stderr || result.stdout).trim()}`, '{cli} down, then {cli} up'); + } + return lease; + }, + + async release(lease) { + const { claim } = readClaim(claimsDir, 'android', lease.slot); + if (claim !== null && claim.nonce === lease.claimNonce) { + await clearSlot(claim, claim.worktree); + } + }, + + async reapable(owner) { + return readClaims(claimsDir, 'android') + .filter((claim) => isOrphaned(claim) || (owner !== undefined && claim.worktree === owner)) + .map((claim) => ({ + backend: 'local' as const, + platform: 'android' as const, + slot: claim.slot, + deviceName: claim.deviceName, + deviceId: laneSerial(claim.slot), + claimNonce: claim.nonce, + acquiredAt: claim.createdAt, + installedBuild: null, + })); + }, + + async startRecording(lease, into) { + return startScreenrecord({ adbBin, serial: lease.deviceId, into }); + }, + + async logs(lease, since, extraPredicate) { + const result = await adb(lease.deviceId, logcatArgs(since, extraPredicate)); + return result.stdout; + }, + + describe: (lease) => `${lease.deviceName} (${lease.deviceId})`, + + async doctorChecks() { + const device: DoctorCheck[] = []; + const avds = await run(emulatorBin, ['-list-avds']); + const adbVersion = await run(adbBin, ['version']); + const ports = `lanes boot it -read-only on ports ${lanePort(1)} to ${lanePort(LOCAL_POOL.android)}`; + if (avds.code !== 0 || adbVersion.code !== 0) { + device.push({ id: 'template', ok: false, detail: 'the Android SDK emulator or adb does not run', fix: 'install the Android SDK (Android Studio) and set ANDROID_HOME' }); + } else { + device.push({ id: 'template', ok: true, detail: listsAvd(avds.stdout) ? `${AVD_NAME}; ${ports}` : `no ${AVD_NAME} AVD yet; the first up writes it, and ${ports}` }); + } + device.push(await lanePortsCheck()); + return { toolchain: [jdkCheck(machine.env)], device }; + }, + }; + return backend; +} + +interface ScreenrecordOptions { + readonly adbBin: string; + readonly serial: string; + readonly into: EvidencePath; +} + +export async function startScreenrecord(options: ScreenrecordOptions): Promise { + const { adbBin, serial, into } = options; + const shell = async (command: string) => (await run(adbBin, ['-s', serial, 'shell', command])).stdout.trim(); + const remote = recordingOnDevice(basename(into)); + const child = spawn(adbBin, ['-s', serial, ...screenrecordArgs(remote)], { stdio: ['ignore', 'pipe', 'pipe'] }); + const startedAt = Date.now(); + let output = ''; + child.stdout.on('data', (chunk: Buffer) => (output += chunk.toString())); + child.stderr.on('data', (chunk: Buffer) => (output += chunk.toString())); + const done = new Promise((resolve) => child.on('close', (code) => resolve(code ?? 1))); + const deadline = Date.now() + 10_000; + while ((await shell('pidof screenrecord')) === '') { + const exited = await Promise.race([done, sleep(500).then(() => null)]); + if (exited !== null || Date.now() >= deadline) { + child.kill(); + throw new VerifyFailure('NOT_READY', `screenrecord did not start on ${serial}: ${output.trim() || `exit ${exited}`}`, 'rerun with --no-video, or `{cli} down` and `{cli} up`'); + } + } + return { + process: { pid: child.pid ?? 0, startedAt }, + async stop() { + await shell('pkill -INT screenrecord'); + const stopDeadline = Date.now() + 30_000; + while ((await shell('pidof screenrecord')) !== '' && Date.now() < stopDeadline) await sleep(500); + await Promise.race([done, sleep(5000)]); + const video = join(into, 'video.mp4') as EvidencePath; + const pulled = await run(adbBin, ['-s', serial, 'pull', remote, video]); + await shell(`rm -f ${remote}`); + if (pulled.code !== 0 || !existsSync(video)) { + throw new VerifyFailure('NOT_READY', `adb pull of the recording from ${serial} failed: ${(pulled.stderr || pulled.stdout).trim()}`, 'rerun the spec, or rerun with --no-video'); + } + return video; + }, + }; +} diff --git a/integration/expo-native/src/platform/android/sdk.ts b/integration/expo-native/src/platform/android/sdk.ts new file mode 100644 index 00000000000..3be7d47c22d --- /dev/null +++ b/integration/expo-native/src/platform/android/sdk.ts @@ -0,0 +1,170 @@ +import { closeSync, existsSync, mkdirSync, openSync, readFileSync, writeFileSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { delimiter, dirname, join } from 'node:path'; +import type { Availability, DoctorCheck } from '../../core/types.ts'; + +export const AVD_NAME = 'Clerk_Verify_Pixel'; +const IMAGE_API = 36; +const IMAGE_TAG = 'google_apis'; +const MIN_JAVA = 21; + +type Env = Readonly>; + +export interface Machine { + readonly os: NodeJS.Platform; + readonly arch: string; + readonly home: string; + readonly env: Env; + readonly kvm: string; +} + +export const thisMachine = (): Machine => ({ os: process.platform, arch: process.arch, home: homedir(), env: process.env, kvm: '/dev/kvm' }); + +const emulatorIn = (root: string) => join(root, 'emulator', 'emulator'); +const adbIn = (root: string) => join(root, 'platform-tools', 'adb'); + +function sdkCandidates(machine: Machine): readonly string[] { + const fromPath = (machine.env.PATH ?? '') + .split(delimiter) + .filter((entry) => /[\\/](platform-tools|emulator)[\\/]?$/.test(entry)) + .map((entry) => dirname(entry.replace(/[\\/]$/, ''))); + const byDefault = machine.os === 'darwin' ? join(machine.home, 'Library', 'Android', 'sdk') : join(machine.home, 'Android', 'Sdk'); + return [...new Set([machine.env.ANDROID_HOME, machine.env.ANDROID_SDK_ROOT, byDefault, ...fromPath].filter((root): root is string => root !== undefined && root !== ''))]; +} + +export function sdkRoot(machine: Machine = thisMachine()): string { + const candidates = sdkCandidates(machine); + return candidates.find((root) => existsSync(emulatorIn(root)) && existsSync(adbIn(root))) ?? candidates[0]!; +} + +export function sdkTool(tool: 'adb' | 'emulator', machine: Machine = thisMachine()): string { + const path = tool === 'adb' ? adbIn(sdkRoot(machine)) : emulatorIn(sdkRoot(machine)); + return existsSync(path) ? path : tool; +} + +const imageOf = (abi: string): string => `system-images;android-${IMAGE_API};${IMAGE_TAG};${abi}`; +const dirOf = (image: string): string => `${image.split(';').join('/')}/`; + +function abiOf(machine: Machine): string { + const own = machine.arch === 'arm64' ? 'arm64-v8a' : 'x86_64'; + const other = own === 'x86_64' ? 'arm64-v8a' : 'x86_64'; + const installed = machine.os === 'darwin' ? [own, other].find((abi) => existsSync(join(sdkRoot(machine), dirOf(imageOf(abi)), 'system.img'))) : undefined; + return installed ?? own; +} + +export const systemImage = (machine: Machine = thisMachine()): string => imageOf(abiOf(machine)); + +const avdHome = (machine: Machine): string => machine.env.ANDROID_AVD_HOME || join(machine.env.ANDROID_USER_HOME || join(machine.home, '.android'), 'avd'); +const avdPointer = (machine: Machine): string => join(avdHome(machine), `${AVD_NAME}.ini`); + +function laneAvdImageDirInSdk(machine: Machine): string | null { + try { + const dir = /^path\s*=\s*(.+)$/m.exec(readFileSync(avdPointer(machine), 'utf8'))?.[1]?.trim() ?? join(avdHome(machine), `${AVD_NAME}.avd`); + return /^image\.sysdir\.1\s*=\s*(.+)$/m.exec(readFileSync(join(dir, 'config.ini'), 'utf8'))?.[1]?.trim() ?? null; + } catch { + return null; + } +} + +export function ensureLaneAvd(machine: Machine = thisMachine()): 'exists' | 'created' { + if (existsSync(avdPointer(machine))) return 'exists'; + const dir = join(avdHome(machine), `${AVD_NAME}.avd`); + mkdirSync(dir, { recursive: true }); + const abi = abiOf(machine); + const config = { + AvdId: AVD_NAME, + 'PlayStore.enabled': 'false', + 'abi.type': abi, + 'avd.ini.encoding': 'UTF-8', + 'disk.dataPartition.size': '6G', + 'hw.cpu.arch': abi === 'x86_64' ? 'x86_64' : 'arm64', + 'hw.cpu.ncore': '4', + 'hw.gpu.enabled': 'yes', + 'hw.gpu.mode': 'auto', + 'hw.keyboard': 'yes', + 'hw.lcd.density': '480', + 'hw.lcd.height': '2856', + 'hw.lcd.width': '1280', + 'hw.ramSize': '2048', + 'image.sysdir.1': dirOf(imageOf(abi)), + 'tag.display': 'Google APIs', + 'tag.id': IMAGE_TAG, + 'vm.heapSize': '256', + }; + writeFileSync(join(dir, 'config.ini'), Object.entries(config).map(([key, value]) => `${key}=${value}\n`).join('')); + writeFileSync(avdPointer(machine), `avd.ini.encoding=UTF-8\npath=${dir}\npath.rel=avd/${AVD_NAME}.avd\ntarget=android-${IMAGE_API}\n`); + return 'created'; +} + +function opensReadWrite(path: string): boolean { + try { + closeSync(openSync(path, 'r+')); + return true; + } catch { + return false; + } +} + +const KVM_RULE = `echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules && sudo udevadm control --reload-rules && sudo udevadm trigger --name-match=kvm`; + +export function localAvailability(machine: Machine = thisMachine()): Availability { + if (machine.os !== 'darwin' && machine.os !== 'linux') return { usable: false, why: `the lane emulator runs on macOS and Linux, and this machine runs ${machine.os}` }; + if (machine.os === 'linux' && !existsSync(machine.kvm)) return { usable: false, why: `there is no ${machine.kvm}, so this machine has no hardware virtualization for the emulator` }; + const root = sdkRoot(machine); + if (!existsSync(emulatorIn(root)) || !existsSync(adbIn(root))) { + return { + usable: false, + why: `no Android SDK with an emulator and adb (looked in ${sdkCandidates(machine).join(', ')})`, + fix: 'install the Android SDK emulator and platform-tools and set ANDROID_HOME to the SDK', + }; + } + const named = laneAvdImageDirInSdk(machine); + const image = named ?? dirOf(systemImage(machine)); + if (!existsSync(join(root, image, 'system.img'))) { + const wanted = image.replace(/\/$/, '').split('/').join(';'); + return { usable: false, why: `the SDK at ${root} has no system image ${wanted}${named === null ? '' : `, which the ${AVD_NAME} AVD names`}`, fix: `sdkmanager "${wanted}", or install it from Android Studio's SDK Manager` }; + } + if (machine.os === 'linux' && !opensReadWrite(machine.kvm)) { + return { usable: false, why: `this user cannot open ${machine.kvm} for reading and writing, so the emulator would have no hardware acceleration`, fix: KVM_RULE }; + } + return { usable: true, why: machine.os === 'linux' ? `this machine runs the emulator itself: ${machine.kvm} opens for reading and writing and the SDK at ${root} has the system image` : `this Mac runs the emulator itself, from the SDK at ${root}` }; +} + +function javaMajor(home: string): number | null { + try { + const release = readFileSync(join(home, 'release'), 'utf8'); + const version = /^JAVA_VERSION="([^"]+)"/m.exec(release)?.[1]; + if (version === undefined) return null; + const major = Number(version.startsWith('1.') ? version.split('.')[1] : version.split(/[.+-]/)[0]); + return Number.isNaN(major) ? null : major; + } catch { + return null; + } +} + +const STUDIO_JBR = process.platform === 'darwin' ? '/Applications/Android Studio.app/Contents/jbr/Contents/Home' : null; + +export type JavaHome = { readonly ok: true; readonly home: string; readonly detail: string } | { readonly ok: false; readonly detail: string; readonly fix: string }; + +export function resolveJavaHome(env: Env = process.env, studioJbr: string | null = STUDIO_JBR): JavaHome { + const studio = studioJbr === null ? null : javaMajor(studioJbr); + const fix = + studioJbr !== null && studio !== null && studio >= MIN_JAVA + ? `export JAVA_HOME="${studioJbr}"` + : studioJbr === null + ? `install a Java ${MIN_JAVA} JDK and export JAVA_HOME to it` + : `install Android Studio (its bundled JBR at ${studioJbr} is Java ${MIN_JAVA}) or a Java ${MIN_JAVA} JDK, then export JAVA_HOME to it`; + const requested = env.JAVA_HOME; + if (requested !== undefined && requested !== '') { + const major = javaMajor(requested); + if (major !== null && major >= MIN_JAVA) return { ok: true, home: requested, detail: `Java ${major} from JAVA_HOME (${requested})` }; + return { ok: false, detail: `JAVA_HOME is ${major === null ? 'not a JDK' : `Java ${major}`} (${requested}); the Android build needs Java ${MIN_JAVA}`, fix }; + } + if (studioJbr !== null && studio !== null && studio >= MIN_JAVA) return { ok: true, home: studioJbr, detail: `Java ${studio} from the Android Studio JBR` }; + return { ok: false, detail: `JAVA_HOME is unset${studioJbr === null ? '' : ` and there is no Java ${MIN_JAVA} Android Studio JBR`}`, fix }; +} + +export function jdkCheck(env: Env = process.env, studioJbr: string | null = STUDIO_JBR): DoctorCheck { + const java = resolveJavaHome(env, studioJbr); + return java.ok ? { id: 'jdk', ok: true, detail: java.detail } : { id: 'jdk', ok: false, detail: java.detail, fix: java.fix }; +} diff --git a/integration/expo-native/src/platform/ios/local.ts b/integration/expo-native/src/platform/ios/local.ts new file mode 100644 index 00000000000..b2b8fe1f26d --- /dev/null +++ b/integration/expo-native/src/platform/ios/local.ts @@ -0,0 +1,286 @@ +import { spawn } from 'node:child_process'; +import { existsSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { join } from 'node:path'; +import { defaultClaimsDir, freeSlot, isOrphaned, readClaim, readClaims, takeSlot, type Claim } from '../../core/claims.ts'; +import { run, sleep } from '../../core/exec.ts'; +import { + LOCAL_POOL, + VerifyFailure, + type AcquireRequest, + type DeviceBackend, + type DoctorCheck, + type EvidencePath, + type LocalLease, + type Recording, +} from '../../core/types.ts'; + +import { recordVideo, showLogs } from './simulator.ts'; + +const TEMPLATE_NAME = 'Clerk Verify Template iOS'; +const LANE_NAME = /^verify-ios-(\d+)$/; + +interface Simulator { + readonly udid: string; + readonly name: string; + readonly state: string; + readonly runtime: string; +} + +async function listSimulators(): Promise { + const listed = await run('xcrun', ['simctl', 'list', 'devices', '-j']); + if (listed.code !== 0) throw new VerifyFailure('NOT_READY', `simctl list failed: ${listed.stderr.trim()}`, 'install Xcode and run `xcode-select -p`'); + const parsed = JSON.parse(listed.stdout) as { devices: Record }; + return Object.entries(parsed.devices).flatMap(([runtime, devices]) => devices.map((d) => ({ udid: d.udid, name: d.name, state: d.state, runtime }))); +} + +async function simctl(args: readonly string[], what: string): Promise { + const result = await run('xcrun', ['simctl', ...args]); + if (result.code !== 0) throw new VerifyFailure('NOT_READY', `${what} failed: ${result.stderr.trim() || result.stdout.trim()}`, 'run `{cli} doctor`'); + return result.stdout; +} + +function simulatorDataDir(udid: string): string { + return join(homedir(), 'Library', 'Developer', 'CoreSimulator', 'Devices', udid, 'data'); +} + +async function deleteSimulators(match: (device: Simulator) => boolean): Promise { + for (const device of (await listSimulators()).filter((d) => LANE_NAME.test(d.name) && match(d))) { + if (device.state !== 'Shutdown') await run('xcrun', ['simctl', 'shutdown', device.udid]); + await simctl(['delete', device.udid], `simctl delete ${device.name}`); + } +} + +export function localIosBackend(options: { readonly os?: NodeJS.Platform } = {}): DeviceBackend { + const claimsDir = defaultClaimsDir(); + const os = options.os ?? process.platform; + + async function claimSlot(request: AcquireRequest): Promise { + const deadline = Date.now() + request.waitSeconds * 1000; + let lastWait = ''; + for (;;) { + const devices = await listSimulators(); + const claims = readClaims(claimsDir, 'ios'); + const orphans = new Set(claims.filter(isOrphaned).map((c) => c.deviceName as string)); + const bootedLanes = devices.filter((d) => d.runtime.includes('iOS') && d.state === 'Booted' && d.name.startsWith('verify-')).map((d) => d.name); + const inUse = new Set([...bootedLanes, ...claims.map((c) => c.deviceName as string)].filter((name) => !orphans.has(name))); + if (inUse.size < LOCAL_POOL.ios) { + for (let slot = 1; slot <= LOCAL_POOL.ios; slot += 1) { + const { gen, claim: holder } = readClaim(claimsDir, 'ios', slot); + if (holder !== null && !isOrphaned(holder)) continue; + const claim = takeSlot(claimsDir, 'ios', slot, gen, request.worktree); + if (claim !== null) return claim; + } + } + if (Date.now() >= deadline) { + throw new VerifyFailure( + 'POOL_FULL', + `${inUse.size} of ${LOCAL_POOL.ios} iOS lanes are in use on this Mac (${[...inUse].sort().join(', ')})`, + `rerun with a longer --wait than ${request.waitSeconds}s, for example ${request.retryWith.replace('', String(Math.max(600, request.waitSeconds * 2)))}, or run {cli} down in a worktree that no longer needs its lane`, + ); + } + const changing = LOCAL_POOL.ios - inUse.size; + const names = `${[...inUse].sort().join(', ')}${changing > 0 ? `, and ${changing} changing hands` : ''}`; + const waiting = `wait no free iOS lane of ${LOCAL_POOL.ios} (${names}); waiting up to ${request.waitSeconds}s for one`; + if (waiting !== lastWait) request.progress(waiting); + lastWait = waiting; + await sleep(5000); + } + } + + async function clearSlot(claim: Claim, worktree: string): Promise { + const reaper = takeSlot(claimsDir, 'ios', claim.slot, claim.gen, worktree, true); + if (reaper === null) return; + await deleteSimulators((d) => d.name === claim.deviceName); + freeSlot(claimsDir, reaper); + } + + const backend: DeviceBackend = { + kind: 'local', + platform: 'ios', + availability: () => (os === 'darwin' ? { usable: true, why: 'this Mac runs the simulator itself' } : { usable: false, why: `the iOS simulator needs macOS and this machine runs ${os}` }), + requirement: 'a Mac with Xcode', + + async acquire(request) { + const template = (await listSimulators()).find((d) => d.name === TEMPLATE_NAME); + if (template === undefined) { + throw new VerifyFailure('NOT_READY', `no simulator named ${TEMPLATE_NAME}`, `xcrun simctl clone "iPhone Air" "${TEMPLATE_NAME}"`); + } + if (template.state !== 'Shutdown') { + throw new VerifyFailure('NOT_READY', `${TEMPLATE_NAME} is ${template.state}; it can only be cloned while shut down`, `xcrun simctl shutdown "${TEMPLATE_NAME}"`); + } + const claim = await claimSlot(request); + let udid = ''; + try { + await deleteSimulators((d) => d.name === claim.deviceName); + request.progress(`device ${claim.deviceName} cloning ${TEMPLATE_NAME}`); + udid = (await simctl(['clone', template.udid, claim.deviceName], `simctl clone ${claim.deviceName}`)).trim(); + await simctl(['boot', udid], `simctl boot ${claim.deviceName}`); + await simctl(['bootstatus', udid, '-b'], `simctl bootstatus ${claim.deviceName}`); + } catch (error) { + await clearSlot(claim, request.worktree).catch(() => undefined); + throw error; + } + return { + backend: 'local', + platform: 'ios', + slot: claim.slot, + deviceName: claim.deviceName, + deviceId: udid, + claimNonce: claim.nonce, + acquiredAt: new Date().toISOString(), + installedBuild: null, + }; + }, + + async check(lease) { + if (readClaim(claimsDir, 'ios', lease.slot).claim?.nonce !== lease.claimNonce) return 'lost'; + const device = (await listSimulators()).find((d) => d.udid === lease.deviceId); + if (device === undefined || device.name !== lease.deviceName) return 'lost'; + if (device.state !== 'Booted') { + await simctl(['boot', lease.deviceId], `simctl boot ${lease.deviceName}`).catch(() => undefined); + await simctl(['bootstatus', lease.deviceId, '-b'], `simctl bootstatus ${lease.deviceName}`); + } + return 'held'; + }, + + async install(lease, app) { + if (app.path === null) throw new VerifyFailure('BUILD_FAILED', `build ${app.key} was made on another machine, so there is no app here to install on ${lease.deviceName}`, '{cli} down, then {cli} up'); + await simctl(['install', lease.deviceId, app.path], `simctl install on ${lease.deviceName}`); + return lease; + }, + + async release(lease) { + const { claim } = readClaim(claimsDir, 'ios', lease.slot); + if (claim !== null && claim.nonce === lease.claimNonce) { + await clearSlot(claim, claim.worktree); + return; + } + if (lease.deviceId !== '') await deleteSimulators((d) => d.udid === lease.deviceId); + }, + + async reapable(owner) { + return readClaims(claimsDir, 'ios') + .filter((claim) => isOrphaned(claim) || (owner !== undefined && claim.worktree === owner)) + .map((claim) => ({ + backend: 'local' as const, + platform: 'ios' as const, + slot: claim.slot, + deviceName: claim.deviceName, + deviceId: '', + claimNonce: claim.nonce, + acquiredAt: claim.createdAt, + installedBuild: null, + })); + }, + + async startRecording(lease, into) { + const file = join(into, 'video.mp4') as EvidencePath; + const record = recordVideo(lease.deviceId, file); + const child = spawn(record.command, [...record.args], { stdio: ['ignore', 'pipe', 'pipe'] }); + const spawnedAt = Date.now(); + const exited = new Promise((resolve) => child.on('close', (code) => resolve(code ?? 1))); + await new Promise((resolve, reject) => { + let seen = ''; + const timer = setTimeout(resolve, 10_000); + const onData = (chunk: Buffer) => { + seen += chunk.toString(); + if (/recording started/i.test(seen)) { + clearTimeout(timer); + resolve(); + } + }; + child.stdout.on('data', onData); + child.stderr.on('data', onData); + child.on('close', (code) => { + clearTimeout(timer); + reject(new VerifyFailure('NOT_READY', `simctl recordVideo exited ${code}: ${seen.trim()}`, 'rerun with --no-video, or `{cli} down` and `{cli} up`')); + }); + }); + const recording: Recording = { + process: { pid: child.pid ?? 0, startedAt: spawnedAt }, + async stop() { + child.kill('SIGINT'); + let timer: NodeJS.Timeout | undefined; + const code = await Promise.race([exited, new Promise((resolve) => (timer = setTimeout(() => resolve(null), 30_000)))]); + clearTimeout(timer); + if (code === null) { + child.kill('SIGKILL'); + await exited; + } + return file; + }, + }; + return recording; + }, + + async logs(lease, since, extraPredicate) { + const show = showLogs(lease.deviceId, since, extraPredicate ?? null); + return (await run(show.command, show.args)).stdout; + }, + + describe: (lease) => lease.deviceName, + + async doctorChecks() { + const xcode = await run('xcodebuild', ['-version']); + const toolchain: DoctorCheck[] = [ + xcode.code === 0 + ? { id: 'xcode', ok: true, detail: xcode.stdout.split('\n')[0]?.replace('Xcode ', '') ?? '' } + : { id: 'xcode', ok: false, detail: 'xcodebuild is not available', fix: 'install Xcode and run `sudo xcode-select -s /Applications/Xcode.app`' }, + ]; + const template = xcode.code === 0 ? (await listSimulators()).find((d) => d.name === TEMPLATE_NAME) : undefined; + const device: DoctorCheck[] = [ + template === undefined + ? { id: 'template', ok: false, detail: `no simulator named ${TEMPLATE_NAME}`, fix: `xcrun simctl clone "iPhone Air" "${TEMPLATE_NAME}", then boot it once, trust your proxy CA, and shut it down` } + : template.state === 'Shutdown' + ? { id: 'template', ok: true, detail: `${TEMPLATE_NAME} (${template.runtime.replace(/^.*SimRuntime\./, '')})` } + : { id: 'template', ok: false, detail: `${TEMPLATE_NAME} is ${template.state}; lanes clone it only while it is shut down`, fix: `xcrun simctl shutdown "${TEMPLATE_NAME}"` }, + await proxyTrustCheck(template), + await lanePortsCheck(claimsDir), + ]; + return { toolchain, device }; + }, + }; + return backend; +} + +const TRUST_STORE_LOCATIONS = [ + ['private', 'var', 'protected', 'trustd', 'private', 'TrustStore.sqlite3'], + ['Library', 'Keychains', 'TrustStore.sqlite3'], +]; + +async function proxyTrustCheck(template: Simulator | undefined): Promise { + const proxy = await run('scutil', ['--proxy']); + const httpsEnabled = /HTTPSEnable\s*:\s*1/.test(proxy.stdout); + if (!httpsEnabled) return { id: 'proxy-trust', ok: true, detail: 'no system HTTPS proxy' }; + const where = /HTTPSProxy\s*:\s*(\S+)/.exec(proxy.stdout)?.[1] ?? 'unknown'; + const port = /HTTPSPort\s*:\s*(\d+)/.exec(proxy.stdout)?.[1] ?? ''; + const proxyName = `${where}${port ? `:${port}` : ''}`; + if (template === undefined) return { id: 'proxy-trust', ok: false, detail: `HTTPS proxy ${proxyName} is on and there is no template to check`, fix: 'create the template first (see the template check)' }; + const store = TRUST_STORE_LOCATIONS.map((parts) => join(simulatorDataDir(template.udid), ...parts)).find((path) => existsSync(path)); + if (store === undefined) { + return { id: 'proxy-trust', ok: false, detail: `HTTPS proxy ${proxyName} is on and ${TEMPLATE_NAME} has no trust store`, fix: `boot ${TEMPLATE_NAME}, install and trust the proxy CA, then shut it down` }; + } + const rows = await run('sqlite3', [store, 'select count(*) from tsettings']); + const count = Number(rows.stdout.trim()); + return rows.code === 0 && count >= 1 + ? { id: 'proxy-trust', ok: true, detail: `${TEMPLATE_NAME} trusts ${count} custom CA${count === 1 ? '' : 's'} (proxy ${proxyName})` } + : { id: 'proxy-trust', ok: false, detail: `HTTPS proxy ${proxyName} is on and ${TEMPLATE_NAME} trusts no custom CA`, fix: `boot ${TEMPLATE_NAME}, install and trust the proxy CA, then shut it down` }; +} + +async function lanePortsCheck(claimsDir: string): Promise { + const booted = (await listSimulators()).filter((d) => d.state === 'Booted' && LANE_NAME.test(d.name)); + const claimed = new Set( + readClaims(claimsDir, 'ios') + .filter((c) => !isOrphaned(c)) + .map((c) => c.deviceName as string), + ); + const foreign = booted.filter((d) => !claimed.has(d.name)); + if (foreign.length === 0) return { id: 'lane-ports', ok: true, detail: 'every booted verify-ios- lane has a live claim' }; + return { + id: 'lane-ports', + ok: false, + detail: `booted with no live claim: ${foreign.map((d) => `${d.name} (${d.udid})`).join(', ')}`, + fix: `if it is yours, ${foreign.map((d) => `xcrun simctl shutdown ${d.udid} && xcrun simctl delete ${d.udid}`).join('; ')}`, + }; +} diff --git a/integration/expo-native/src/platform/ios/simulator.ts b/integration/expo-native/src/platform/ios/simulator.ts new file mode 100644 index 00000000000..4a16c2f33fd --- /dev/null +++ b/integration/expo-native/src/platform/ios/simulator.ts @@ -0,0 +1,19 @@ +import type { CommandLine } from '../../core/exec.ts'; + +const LOG_PREDICATE = 'subsystem == "com.clerk.verify" OR subsystem == "com.clerk.sdk"'; + +function localTime(date: Date): string { + const pad = (n: number) => String(n).padStart(2, '0'); + return `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())} ${pad(date.getHours())}:${pad(date.getMinutes())}:${pad(date.getSeconds())}`; +} + +export const waitForBoot = (udid: string): CommandLine => ({ command: 'xcrun', args: ['simctl', 'bootstatus', udid, '-b'] }); + +export const installApp = (udid: string, app: string): CommandLine => ({ command: 'xcrun', args: ['simctl', 'install', udid, app] }); + +export const recordVideo = (udid: string, file: string): CommandLine => ({ command: 'xcrun', args: ['simctl', 'io', udid, 'recordVideo', '--codec=h264', '--force', file] }); + +export const showLogs = (udid: string, since: Date, extraPredicate: string | null): CommandLine => ({ + command: 'xcrun', + args: ['simctl', 'spawn', udid, 'log', 'show', '--style', 'compact', '--start', localTime(since), '--predicate', extraPredicate === null ? LOG_PREDICATE : `${LOG_PREDICATE} OR (${extraPredicate})`], +}); From ebb8a4f37db39b8439830d6d7c45374ed7aa98cb Mon Sep 17 00:00:00 2001 From: Mike Pitre <12040919+mikepitre@users.noreply.github.com> Date: Wed, 7 Oct 2026 20:31:00 -0400 Subject: [PATCH 5/7] test(expo): add the golden tests, their fixtures, and the feature files Co-Authored-By: Claude Opus 5.5 --- .../verify-clerk-expo/features/README.md | 54 +++++++ .../features/custom-flow-sign-in.md | 29 ++++ .../features/custom-flow-sign-up.md | 27 ++++ .../features/native-auth-view.md | 44 ++++++ .../features/native-js-sync.md | 36 +++++ .../features/native-modules.md | 36 +++++ .../features/token-cache-persistence.md | 32 ++++ .../features/user-button-and-profile.md | 39 +++++ integration/expo-native/specs/fixtures.ts | 136 +++++++++++++++++ .../custom-flow-sign-in/complete.e2e.ts | 17 +++ .../custom-flow-sign-up/complete.e2e.ts | 18 +++ .../golden/native-auth-view/complete.e2e.ts | 19 +++ .../specs/golden/native-auth-view/logo.e2e.ts | 28 ++++ .../golden/native-auth-view/opens.e2e.ts | 37 +++++ .../native-js-sync/sign-in-from-native.e2e.ts | 36 +++++ .../sign-out-from-native.e2e.ts | 14 ++ .../biometric-availability.e2e.ts | 18 +++ .../biometric-availability.settings.json | 4 + .../native-modules/google-sign-in.e2e.ts | 33 +++++ .../token-cache-persistence/relaunch.e2e.ts | 30 ++++ .../embedded-profile.e2e.ts | 45 ++++++ .../user-button-and-profile/profile.e2e.ts | 23 +++ integration/expo-native/specs/native.ts | 137 ++++++++++++++++++ integration/expo-native/src/core/MANIFEST | 1 + 24 files changed, 893 insertions(+) create mode 100644 .claude/skills/verify-clerk-expo/features/README.md create mode 100644 .claude/skills/verify-clerk-expo/features/custom-flow-sign-in.md create mode 100644 .claude/skills/verify-clerk-expo/features/custom-flow-sign-up.md create mode 100644 .claude/skills/verify-clerk-expo/features/native-auth-view.md create mode 100644 .claude/skills/verify-clerk-expo/features/native-js-sync.md create mode 100644 .claude/skills/verify-clerk-expo/features/native-modules.md create mode 100644 .claude/skills/verify-clerk-expo/features/token-cache-persistence.md create mode 100644 .claude/skills/verify-clerk-expo/features/user-button-and-profile.md create mode 100644 integration/expo-native/specs/fixtures.ts create mode 100644 integration/expo-native/specs/golden/custom-flow-sign-in/complete.e2e.ts create mode 100644 integration/expo-native/specs/golden/custom-flow-sign-up/complete.e2e.ts create mode 100644 integration/expo-native/specs/golden/native-auth-view/complete.e2e.ts create mode 100644 integration/expo-native/specs/golden/native-auth-view/logo.e2e.ts create mode 100644 integration/expo-native/specs/golden/native-auth-view/opens.e2e.ts create mode 100644 integration/expo-native/specs/golden/native-js-sync/sign-in-from-native.e2e.ts create mode 100644 integration/expo-native/specs/golden/native-js-sync/sign-out-from-native.e2e.ts create mode 100644 integration/expo-native/specs/golden/native-modules/biometric-availability.e2e.ts create mode 100644 integration/expo-native/specs/golden/native-modules/biometric-availability.settings.json create mode 100644 integration/expo-native/specs/golden/native-modules/google-sign-in.e2e.ts create mode 100644 integration/expo-native/specs/golden/token-cache-persistence/relaunch.e2e.ts create mode 100644 integration/expo-native/specs/golden/user-button-and-profile/embedded-profile.e2e.ts create mode 100644 integration/expo-native/specs/golden/user-button-and-profile/profile.e2e.ts create mode 100644 integration/expo-native/specs/native.ts diff --git a/.claude/skills/verify-clerk-expo/features/README.md b/.claude/skills/verify-clerk-expo/features/README.md new file mode 100644 index 00000000000..f0cc06326a6 --- /dev/null +++ b/.claude/skills/verify-clerk-expo/features/README.md @@ -0,0 +1,54 @@ +# @clerk/expo verification map + +This directory is the maintained source for verifying the user-facing behavior of `@clerk/expo` through the `expo-native` test app in `integration/templates/expo-native`. On a local device the test app runs as a Debug dev client that loads its JS, and `packages/expo` with it, from a local Metro server. Read this index before driving the app, then use the matching feature file as the recipe. Every recipe runs through `integration/expo-native/bin/control-clerk-expo` and the golden specs under `specs/golden//`. + +## Test users and sign-in + +`.claude/skills/verify-clerk-expo/SKILL.md` has the rules for signing in. These are the test identities and the identifiers a spec needs. + +- **Emails.** Any address that contains `+clerk_test@` is a test address. Clerk sends no mail and accepts the code below. `host.newEmail()` and `host.seedUser()` mint `verify__+clerk_test@example.com`, new per run. +- **One-time code.** `424242` verifies every email code for a test address. Specs use the constant `CLERK_TEST_CODE`. +- **Passwords.** The standard settings require a password at sign-up. Use a throwaway per run, such as `Verify--Pw1!`. For a password sign-in, seed the user with `host.seedUser({ password: true })` and fill the field with `host.fill(field, user.password!)`. + +| Step | Native AuthView (iOS identifier) | Custom flow (test app testID) | +| ------------------------------------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------- | +| Email field | `clerk.auth.start.identifier` | `verify.customSignIn.emailAddress`, `verify.customSignUp.emailAddress` | +| Switch between email and phone | `clerk.auth.start.identifierSwitcher` | none | +| Phone field | `clerk.auth.start.phoneNumber` | none | +| Continue or send code | `clerk.auth.start.continue` | `verify.customSignIn.sendCode`, `verify.customSignUp.sendCode` | +| Try another method on the email link screen | the `Use another method` text | none | +| Pick a method from the list | `clerk.auth.signIn.alternativeMethod.` | none | +| Sign-in code field | `clerk.auth.signIn.code` | `verify.customSignIn.code`, then `verify.customSignIn.verifyCode` | +| Sign-in password | `clerk.auth.signIn.password`, then `clerk.auth.signIn.continue` | none | +| Sign-up password | `clerk.auth.signUp.password` | `verify.customSignUp.password` | +| Sign-up code field | `clerk.auth.signUp.code` | `verify.customSignUp.code`, then `verify.customSignUp.verifyCode` | +| Close AuthView | `clerk.dismissButton` | none | + +The iOS identifiers come from `Sources/ClerkKitUI/Components/Auth/ClerkAccessibilityIdentifiers.swift` in the clerk-ios release that `packages/expo/ios/ClerkExpo.podspec` pins. The clerk-android release that `@clerk/expo` pins has no test tags, so Android specs find native views by their text. The custom-flow testIDs live in `integration/templates/expo-native/screens/`. + +## Driving conventions + +- Prefer SDK identifiers on iOS and test app testIDs everywhere. `specs/native.ts` holds the per-platform locators for native views that have no Android tag yet. +- Look first in the native view on the spec's path: the AuthView code screen, or the profile and its `Manage account` screen. Use the `Token cache` screen for what the token cache held when the app started. + +## Proof and skip reporting + +- A proof is a passing `run` whose run directory holds `video.mp4`, `screenshots/`, `app.log`, and `e2e/report.json`. The video and the screenshots are the proof on both platforms. +- Name the run id, the platform, and the specs in the PR. Attach the run with `attach --pr `. +- A spec limited to one platform reports as skipped on the other. Say which platform each proof ran on. + +## Feature entry contract + +Each feature file starts with an H1 title and one paragraph describing the user-visible behavior, then exactly four H2 sections in this order: `Sub-features`, `How to get to it (user POV)`, `Driving it with verify`, and `Gotchas`. `Driving it with verify` starts with `Preconditions:` and names the golden specs that prove each sub-feature. A feature file lists only entry points that a spec drives. + +## Features + +- [Native AuthView](./native-auth-view.md) covers opening, closing, and signing in through the native AuthView, and its React Native logo. +- [User button and profile](./user-button-and-profile.md) covers the native UserButton, the profile it opens, the home's sign-out button, and an inline UserProfileView with `onHostBack` and a custom page. +- [Custom sign-in flow](./custom-flow-sign-in.md) covers an email code sign-in built on `useSignIn`. +- [Custom sign-up flow](./custom-flow-sign-up.md) covers an email and password sign-up built on `useSignUp`. +- [Token cache persistence](./token-cache-persistence.md) covers restoring the same session after a relaunch. +- [Native and JS session sync](./native-js-sync.md) covers a native sign-out and a native password sign-in reaching `useAuth`, `useUser`, and `useSession`, and that sign-in surviving a relaunch. +- [Native modules](./native-modules.md) covers `useSignInWithGoogle` opening the native Google sign-in, a cancel of it on iOS, and `useBiometricCredentials` reporting what its native module found on the device. + +Not mapped yet: SSO through `useSSO` (no real OAuth on simulators), a Google or Apple native sign-in with a real account, passkeys, enrolling or using a biometric (no associated domains or Secure Enclave on the simulator), and session tasks in the native views. diff --git a/.claude/skills/verify-clerk-expo/features/custom-flow-sign-in.md b/.claude/skills/verify-clerk-expo/features/custom-flow-sign-in.md new file mode 100644 index 00000000000..86ff960519f --- /dev/null +++ b/.claude/skills/verify-clerk-expo/features/custom-flow-sign-in.md @@ -0,0 +1,29 @@ +# Custom sign-in flow + +An app that builds its own sign-in screen on `useSignIn` sends an email code to an existing user and signs them in with it. + +## Sub-features + +- `complete` verifies the code, finalizes the sign-in, and returns to the home, which shows the user and the session. + +## How to get to it (user POV) + +- Tap `Custom sign-in` on the signed-out home. The screen has an email field, `Send code`, then a code field and `Verify code`. + +## Driving it with verify + +Preconditions: + +- The standard settings turn the `email_code` strategy on. `up` fails with `INSTANCE_MISCONFIGURED` when the application does not show it. +- The spec seeds its own `+clerk_test` user. + +- **Request the code.** Run `integration/expo-native/bin/control-clerk-expo run custom-flow-sign-in/complete`. The spec fills `verify.customSignIn.emailAddress`, taps `verify.customSignIn.sendCode`, and expects `verify.customSignIn.code` with no `verify.customSignIn.error`. The form shows the code field only when `signIn.emailCode.sendCode` returned no error. Screenshot `custom-code`. +- **Enter the code.** The spec then types `CLERK_TEST_CODE` and taps `verify.customSignIn.verifyCode`, and `host.expectSignedInAs(user)` waits for the home to show the user's email, the seeded user ID, and a session ID. Screenshot `custom-signed-in`. +- **Proof.** The spec passes on the platform you changed. + +## Gotchas + +- These are React Native views with `testID`s, so plain locator actions work. `host.tap` and `host.fill` also work and keep specs uniform. On iOS an empty one reads as its placeholder, and `host.fill` replaces its contents once when the field does not hold the text. +- A sign-in error shows in `verify.customSignIn.error`, and the form stays on screen. Read the error text from the failure page when a spec stops on the email screen. +- The test app shows the form only while nobody has an active session. Once the sign-in is finalized it shows the home, so the form on screen also means that nobody is signed in yet. +- The screen keeps no state across launches. Every launch starts on the home, and the form opens on the email field. diff --git a/.claude/skills/verify-clerk-expo/features/custom-flow-sign-up.md b/.claude/skills/verify-clerk-expo/features/custom-flow-sign-up.md new file mode 100644 index 00000000000..2e62001ca22 --- /dev/null +++ b/.claude/skills/verify-clerk-expo/features/custom-flow-sign-up.md @@ -0,0 +1,27 @@ +# Custom sign-up flow + +An app that builds its own sign-up screen on `useSignUp` creates a user with an email address and password, then verifies the email code. + +## Sub-features + +- `complete` verifies the code, finalizes the sign-up, and returns to the home, which shows the new user and the session. + +## How to get to it (user POV) + +- Tap `Custom sign-up` on the signed-out home. The screen has email and password fields, `Send code`, then a code field and `Verify code`. + +## Driving it with verify + +Preconditions: + +- The standard settings require a password at sign-up. The spec uses `Verify--Pw1!`. +- The spec gets a new `+clerk_test` address from `host.newEmail`. The address starts with the run's prefix, and that is how the run finds the user the form creates. `integration/expo-native/bin/control-clerk-expo down` deletes the application and with it the user the form creates. + +- **Request the code.** Run `integration/expo-native/bin/control-clerk-expo run custom-flow-sign-up/complete`. The spec fills `verify.customSignUp.emailAddress` and `verify.customSignUp.password`, taps `verify.customSignUp.sendCode`, and expects `verify.customSignUp.code` with no `verify.customSignUp.error`. The form shows the code field only when `signUp.create` and `sendEmailCode` returned no error. Screenshot `custom-signup-code`. +- **Enter the code.** The spec then types `CLERK_TEST_CODE` and taps `verify.customSignUp.verifyCode`, and `host.expectSignedInAs(email)` waits for the home to show the new email, a user ID, and a session ID. Screenshot `custom-signed-up`. +- **Proof.** The spec shows the home with the new email, a user ID, and a session ID. + +## Gotchas + +- Type only an address that `host.newEmail` returned, so the run finds the identity it created. +- The test app's password field has the one-time-code content type (`textContentType='oneTimeCode'` in `screens/CustomSignUp.tsx`). Without it, iOS covers the keyboard with a `Use Strong Password?` sheet as soon as the field has focus. With it, a spec that only focuses the field sees the keyboard and no sheet. The specs that type the password pass with it. diff --git a/.claude/skills/verify-clerk-expo/features/native-auth-view.md b/.claude/skills/verify-clerk-expo/features/native-auth-view.md new file mode 100644 index 00000000000..e187a9e7595 --- /dev/null +++ b/.claude/skills/verify-clerk-expo/features/native-auth-view.md @@ -0,0 +1,44 @@ +# Native AuthView + +A signed-out user sees the native AuthView (SwiftUI from clerk-ios on iOS, Compose from clerk-android on Android) inside the Expo app, starts sign-in with an email address, and can close the view to get back to the JS screen. An app can give the view a React Native logo. + +## Sub-features + +- `opens` shows the AuthView start screen as the root of the window from the home's `Sign in full screen` button, once `useAuthViewState().isLoaded` is true, on iOS and Android. +- `opens-from-home` opens AuthView in a modal from the home's `Sign in` button, on iOS and Android. +- `close-modal` closes the AuthView that the home opened, with the close button on iOS and the back button on Android, and the home is back. +- `logo` shows a React Native view as the logo of AuthView, and shows it again after AuthView closes and reopens, on iOS and Android. +- `logo-sign-in` signs in with a password through the AuthView that has the logo, after it closed and reopened, on iOS and Android. +- `dismiss` closes a full-screen AuthView with its close button and returns to the home (iOS). +- `complete` moves an existing user from the identifier field to the email code screen, accepts the test code, and signs the user in, so the home shows the user and the session (iOS). + +## How to get to it (user POV) + +- On the signed-out home, tap `Sign in` (AuthView in a modal), `Sign in with a logo` (the same modal with a React Native logo), `Sign in full screen` (AuthView with no close button), or `Sign in full screen with a close button`. +- Type the email address and tap Continue. On the standard settings the first factor is an email link, so tap `Use another method`, then `Email code to
`. + +## Driving it with verify + +Preconditions: + +- `integration/expo-native/bin/control-clerk-expo doctor --platform ` passes apart from `build`. +- `up` has created the worktree's throwaway Clerk application. The sign-in specs seed their own `+clerk_test` user. + +- **Opens.** Run `integration/expo-native/bin/control-clerk-expo run native-auth-view/opens --platform ios`, and again with `--platform android`. The first test taps `Sign in full screen` on the home and expects the identifier field, and on iOS no close button. The test app shows its spinner in place of that AuthView until `useAuthViewState().isLoaded` is true, so the field on screen proves the flag. Screenshot `auth-full-screen`. +- **Opens from the home and closes.** The second test expects the home to be signed out, taps `Sign in`, and expects the identifier field. Screenshot `auth-from-home`. It then closes AuthView with `closeNativeAuth` from `specs/native.ts`, and expects the home to show `Signed out` and the `Sign in` button, with no identifier field left. Screenshot `closed-modal`. +- **Logo.** Run `integration/expo-native/bin/control-clerk-expo run native-auth-view/logo --platform ios`, and again with `--platform android`. It seeds a user with a password, taps `Sign in with a logo` on the home, expects the text `E2E Custom Logo` above the identifier field, closes AuthView, expects the logo to be gone, opens AuthView again, and expects the logo again. Screenshots `logo` and `logo-reopened`. It then signs in with `signInWithPassword` from `specs/native.ts`, and expects the home to show the user. Screenshot `signed-in-with-logo`. +- **Dismiss.** The third test in `opens.e2e.ts` (iOS only) taps `Sign in full screen with a close button` on the home, taps `clerk.dismissButton`, and expects the home to show `Signed out` and the `Sign in` button. While nobody is signed in, the test app shows the home again only when `onDismiss` fires. Screenshot `dismissed-home`. +- **Request the code.** Run `integration/expo-native/bin/control-clerk-expo run native-auth-view/complete` (iOS only). It taps `Sign in full screen`, AuthView with no close button, so nothing can dismiss the form mid-spec. The spec fills `clerk.auth.start.identifier`, taps `clerk.auth.start.continue`, waits for `Use another method`, taps it and `clerk.auth.signIn.alternativeMethod.email_code`, and expects `clerk.auth.signIn.code` and the user's email on the code screen. Screenshot `code-screen`. +- **Enter the code.** The spec then types `CLERK_TEST_CODE`, and `host.expectSignedInAs(user)` waits for the home to show the user's email, the seeded user ID, and a session ID. Screenshot `signed-in`. +- **Proof.** The run directory holds `video.mp4` and `screenshots/auth-full-screen.png`, which shows AuthView with the whole window. + +## Gotchas + +- On iOS a full-screen AuthView renders inline, where SwiftUI's `dismiss` has no effect, so `@clerk/expo` calls `onDismiss` from the close button itself. The `dismiss` test covers that. The close button of the AuthView in the modal goes through the modal's own dismissal, which `close-modal` covers. +- clerk-ios AuthView remembers the last identifier in UserDefaults. The test app clears it on every new storage scope, so the field starts empty. A prefilled field shows as two `clerk.auth.start.identifier` nodes (label and textbox) and fails a locator with `LOCATOR_AMBIGUOUS`. If that happens, the test app's scope clearing regressed. +- `@clerk/expo` ships the clerk-ios release pinned in `packages/expo/ios/ClerkExpo.podspec`, so iOS has the `clerk.auth.*` identifiers. The clerk-android release it pins has no test tags yet, so Android specs find AuthView by its text (`Enter your email or username`, `Continue`). Replace them with `getByTestId` after a clerk-android release with tags lands in `@clerk/expo`. +- The test app puts nothing above, below, or over AuthView. On iOS the screen tree leaves out what a modal covers, so the home's nodes are not in the tree while the modal AuthView is up. +- On Android, an AuthView that can be dismissed shows a close button with the label `Close`, and `Sign in full screen` shows none. No spec taps that button. The back button closes the AuthView that the home opened. +- The `complete` spec is iOS only. On Android, with the clerk-android 1.1.11 that `@clerk/expo` pins, the `Email code to
` row under `Use another method` leads to the email link screen (`Open email app`), which has no code field. The Android sign-in through AuthView is `native-js-sync/sign-in-from-native`, which uses a password. +- Close the modal AuthView with `closeNativeAuth` from `specs/native.ts`. On iOS one tap on the close button can do nothing, and `closeNativeAuth` taps the button once more when it is still on screen three seconds after the first tap. +- The logo is a React Native view that AuthView hosts. The spec finds it by its text on both platforms. diff --git a/.claude/skills/verify-clerk-expo/features/native-js-sync.md b/.claude/skills/verify-clerk-expo/features/native-js-sync.md new file mode 100644 index 00000000000..4dcabaf0be4 --- /dev/null +++ b/.claude/skills/verify-clerk-expo/features/native-js-sync.md @@ -0,0 +1,36 @@ +# Native and JS session sync + +A session change in the native UI reaches the `@clerk/expo` JS hooks, and a session started in JS reaches the native views. + +## Sub-features + +- `native-sign-out` signs out from the native profile, and `useAuth`, `useUser`, and `useSession` report signed out. +- `native-sign-in` signs in with a password in the AuthView that the home opened, and the three hooks report the user and a session. The same session is back after a relaunch. +- `second-native-sign-in` signs in natively, signs out in the native profile, and signs in natively again in the same process, and the hooks report the user again. +- `js-sign-in` signs in with a ticket through `useSignIn`, and the native profile shows the same user. `user-button-and-profile/profile` proves it. +- `js-sign-out` signs out through `useAuth().signOut()`, and the home shows no user and no session. The second test of `user-button-and-profile/profile` proves it. + +## How to get to it (user POV) + +- Tap the UserButton on the home, then tap `Sign out` in the native profile. +- Tap `Sign in` on the home, type the email address, tap Continue, type the password, and tap Continue. + +## Driving it with verify + +Preconditions: + +- Each spec seeds a `+clerk_test` user. The sign-out spec starts signed in through a ticket, because it tests no sign-in. The sign-in spec starts signed out and signs in through the real form. + +- **Native sign-out.** Run `integration/expo-native/bin/control-clerk-expo run native-js-sync/sign-out-from-native --platform ios`, and again with `--platform android`. The spec launches signed in, taps the UserButton, taps `Sign out` in the profile, and calls `host.expectSignedOut`. That waits for the home to show `Signed out` (from `useAuth`), with no `Signed in as` text and no user ID (from `useUser`) and no session ID (from `useSession`). Screenshots `before-sign-out` and `signed-out`. +- **Native sign-in.** Run `integration/expo-native/bin/control-clerk-expo run native-js-sync/sign-in-from-native --platform ios`, and again with `--platform android`. Both tests seed a user with `host.seedUser({ password: true })`, tap `Sign in` on the home, and sign in with `signInWithPassword` from `specs/native.ts`. The first test expects the home to show the user and a session ID, relaunches with `keepStorage`, and expects the same user and the same session ID. Screenshots `signed-in` and `restored-home`. The second test signs out with the profile's `Sign out` row, signs in again, and expects the home to show the user. Screenshot `signed-in-again`. +- **Proof.** `screenshots/signed-out.png` shows the home with `Signed out` and the `Sign in` button. The native profile closes by itself, because the UserButton that presented it leaves the home. + +## Gotchas + +- The `Sign out` row is at the bottom of the profile. On a small simulator it can be below the fold. The devices the CLI leases show it without scrolling. +- A seeded password user is created with `bypass_client_trust`, so the password is the only step of the sign-in. +- AuthView remembers the last identifier. When it opens a second time in one process the email field is already filled, on iOS and Android, and on iOS `clerk.auth.start.identifier` then matches two nodes. `reachPasswordScreen` in `specs/native.ts` types the email only when the field does not show it. +- The password field has no identifier of its own on Android, and on iOS three nodes share `clerk.auth.signIn.password`. The spec fills the last text field on the password screen. +- After a password sign-in iOS can offer to save the password, and `host.expectSignedInAs` declines. On Android the Google Password Manager can offer the same, and `signInWithPassword` declines. +- A sign-out that only clears the native session leaves the user or the session on the home. `host.expectSignedOut` then fails and says what the home shows. That is the bug this feature guards against. +- Each part of the home reads its own hook, so do not merge them. If the home took the session ID from `useAuth`, a stale `useSession` would pass unseen. diff --git a/.claude/skills/verify-clerk-expo/features/native-modules.md b/.claude/skills/verify-clerk-expo/features/native-modules.md new file mode 100644 index 00000000000..3602a573606 --- /dev/null +++ b/.claude/skills/verify-clerk-expo/features/native-modules.md @@ -0,0 +1,36 @@ +# Native modules + +An app that uses `useSignInWithGoogle` from `@clerk/expo/google` or `useBiometricCredentials` from `@clerk/expo/biometrics` gets an answer that comes from native code. The Google hook opens the native Google sign-in of `@clerk/expo-google-signin`, and a user who cancels it is left signed out with no error. The biometrics hook reports what the native module of `@clerk/expo-biometrics` found on the device. + +## Sub-features + +- `google-sign-in` starts the Google flow with `startGoogleAuthenticationFlow()`, and on iOS cancels it. The hook configures the native module with the test app's placeholder client IDs and calls its `presentExplicitSignIn`. On iOS the system asks whether the app may use `accounts.google.com` to sign in. On Android Google Play services opens a page of its own. Cancelling resolves the hook with no session, which the test checks on iOS only. +- `biometric-availability` asks `getAvailability()` whether a biometric credential on this device can sign in. With biometric sign-in on for the instance, the hook calls the native `getAvailability` and shows the reason that follows from its answer. + +## How to get to it (user POV) + +- Tap `Native modules` on the signed-out home. +- Tap `Sign in with Google`. On iOS, tap `Cancel` on the system prompt, and `Google sign-in was cancelled` shows under the button. On Android a page of Google Play services covers the test app. +- Tap `Check biometric availability`. The result shows as text under the button. + +## Driving it with verify + +Preconditions: + +- `up` has built the test app. The build links `@clerk/expo-google-signin` and `@clerk/expo-biometrics` from the workspace. The test app's `app.json` lists the `@clerk/expo-google-signin` config plugin and holds three placeholder values under `extra`: a web client ID, an iOS client ID, and the iOS URL scheme that the plugin registers. They belong to no Google project. +- `specs/golden/native-modules/biometric-availability.settings.json` turns biometric sign-in on for the instance (`auth_biometric.used_for_sign_in`), and `run` applies it before the test of that spec file. The Google spec file runs on the standard settings, so a run of both has two settings groups. + +- **Google sign-in.** Run `integration/expo-native/bin/control-clerk-expo run native-modules/google-sign-in --platform ios`, and again with `--platform android`. The test taps `Native modules` on the home and taps `google-sign-in-button`. On iOS it waits for the system prompt that names `accounts.google.com` and dismisses it with `device.alert('dismiss')`. It then expects `google-result` to say `Google sign-in was cancelled`. On Android it waits for a node whose id begins `com.google.android.gms:id/`, which is a page of Google Play services, and ends there. Screenshots `google-system-prompt` and `google-cancelled`, both on iOS; on Android the video shows Google's page. +- **Biometric availability.** Run `run native-modules/biometric-availability` on each platform. The test opens the same screen, taps `biometric-availability-button`, and expects `biometric-availability-result` to say `biometric availability: biometric_authentication_unavailable` on iOS and `biometric availability: no_local_credential` on Android. Screenshot `biometric-availability`. +- **Proof.** Each screenshot shows the result text under its button. Both tests pass on iOS and Android. + +## Gotchas + +- Neither test signs anyone in. No Google account and no enrolled biometric is involved, and the Google test never loads a Google sign-in form. On iOS it cancels at the system prompt, before any page loads. +- The Google test fails when the native module is not linked, because the hook then throws `ClerkGoogleSignIn native module is not available` and nothing opens. It also fails when the plugin did not register the URL scheme. Google's iOS SDK then raises an exception and the app stops. +- The Google test proves that the hook reaches the native module and that the module opens Google's sign-in. On iOS it also proves that the module reports a cancel. It does not prove that a Google account can sign in, or anything about the ID token and the Clerk sign-in that follow. +- On Android the test does not cancel. On an emulator with no Google account, Google Play services can open a "Sign in with ease" page that the back button does not close, and `Skip` on it leads to an account page whose back button returns to it. Which page opens differs from run to run, so a test that cancels there fails for reasons that are Google's. +- The two biometric answers come from native code. On an iOS simulator the native module reports no Secure Enclave, so the hook answers `biometric_authentication_unavailable`. An Android emulator has key storage, so the hook goes on to the native list of stored credentials, finds none, and answers `no_local_credential`. With the standard settings the hook would answer `feature_disabled` from JS, and the test would fail. +- Neither answer changes when a biometric is enrolled on the device. Enrollment matters only once a credential is stored, and no test stores one. An iOS simulator cannot create the key, and an Android emulator would need a fingerprint and a screen lock set up in each run. +- The buttons and their results have the testIDs `google-sign-in-button`, `google-result`, `biometric-availability-button`, and `biometric-availability-result`, without the `verify.` prefix of the test app's other screens. +- A result that starts with `Google sign-in failed:` or `biometric availability failed:` means the call threw, and the test fails on it. diff --git a/.claude/skills/verify-clerk-expo/features/token-cache-persistence.md b/.claude/skills/verify-clerk-expo/features/token-cache-persistence.md new file mode 100644 index 00000000000..d8745ef0c07 --- /dev/null +++ b/.claude/skills/verify-clerk-expo/features/token-cache-persistence.md @@ -0,0 +1,32 @@ +# Token cache persistence + +A signed-in user who closes and reopens the app is still signed in, because `@clerk/expo`'s token cache restored the client from secure storage. + +## Sub-features + +- `relaunch` restores the same user and the same session after a cold relaunch with no ticket. +- `kept-token` shows that the JS token cache held the client token when the app started. +- `new-scope` starts signed out when storage was cleared, which proves the restore came from storage. + +## How to get to it (user POV) + +- Sign in, close the app, and open it again. The home shows the same user and session ID. Tap `Token cache` on the home. That screen says whether the token cache held a client token when the app started. + +## Driving it with verify + +Preconditions: + +- The spec seeds a `+clerk_test` user and signs in with a ticket on the first launch. + +- **Relaunch.** Run `integration/expo-native/bin/control-clerk-expo run token-cache-persistence --platform android` (or `ios`). The spec launches signed in and reads the session ID on the home. It relaunches with `keepStorage: true` and no ticket, and expects the home to show the same user and the same session ID. Screenshot `restored-home`. +- **Kept token.** On that relaunched home the test taps `Token cache` and expects `client token kept from the last launch: yes` and `user: `. Screenshot `token-cache`. +- **New scope.** The same test relaunches with a new storage scope, taps `Token cache` on the signed-out home, and expects `client token kept from the last launch: no` and `user: none`. Screenshot `new-scope`. +- **Proof.** `screenshots/restored-home.png` shows the user and the session after a launch that had no ticket, and `screenshots/token-cache.png` shows the kept token. + +## Gotchas + +- `keepStorage: true` reuses the previous launch's `verifyStorageScope`. Without it, the test app clears Clerk's stored client on launch, so the relaunch would start signed out. +- On iOS the test app clears every generic-password keychain item the test app owns when the scope changes, not only Clerk's. +- The home alone does not prove the JS token cache. Native storage could restore the session and sync it to JS, and the home would look the same. The `Token cache` screen shows what the JS cache itself held. +- The test app reads the cache once, when the app starts and before Clerk has loaded, and the `Token cache` screen shows that answer whenever the home opens it. Do not read the cache again after Clerk loads. A signed-out client stores a client token of its own by then, so the text would say `yes` on a new scope. +- A launch with no `signedInAs` that keeps storage needs `landsOn`, because only the spec knows that it expects a signed-in home. diff --git a/.claude/skills/verify-clerk-expo/features/user-button-and-profile.md b/.claude/skills/verify-clerk-expo/features/user-button-and-profile.md new file mode 100644 index 00000000000..3c3c94f2bcb --- /dev/null +++ b/.claude/skills/verify-clerk-expo/features/user-button-and-profile.md @@ -0,0 +1,39 @@ +# User button and profile + +A signed-in user opens the native profile from the native UserButton, sees their own account, and can sign out from the app's own button. An app can also render the profile inline, add a page of its own to it, and close it from the profile's Back button. + +## Sub-features + +- `user-button` opens the native profile from the UserButton on the home. +- `profile` shows the profile of the signed-in user, and `Manage account` lists the user's email. +- `home-sign-out` ends the session with the home's `Sign out` button, which calls `useAuth().signOut()`. +- `embedded-profile` renders UserProfileView inline with `isDismissible={false}`. The Back button pops a screen inside the profile, and on the root of the profile it calls `onHostBack`, which shows the home. +- `custom-page` adds a row to the profile with `customPages`, and the row opens a page with React Native content. + +## How to get to it (user POV) + +- Tap the avatar UserButton on the signed-in home. +- Tap `Sign out` on the home. +- Tap `Embedded profile` on the signed-in home. The profile is the root of the window. + +## Driving it with verify + +Preconditions: + +- The spec seeds a `+clerk_test` user and signs in with a ticket (`host.launch({ signedInAs })`). + +- **User button and profile.** Run `integration/expo-native/bin/control-clerk-expo run user-button-and-profile --platform ios`, and again with `--platform android`. The first test launches signed in. The launch lands on the home only when it shows the seeded user's email and user ID and a session ID. The test taps the UserButton, expects `Manage account`, taps it, and expects the user's email. Screenshots `user-button-profile` and `profile`. +- **Home sign-out.** The second test launches signed in, taps `Sign out`, and expects the home to show `Signed out`, no user ID, no session ID, and the `Sign in` button. Screenshot `signed-out`. +- **Embedded profile and `onHostBack`.** Run `integration/expo-native/bin/control-clerk-expo run user-button-and-profile/embedded-profile --platform ios`, and again with `--platform android`. The first test launches signed in, taps `Embedded profile` on the home, and expects `Manage account`. It taps `Manage account`, expects the user's email, taps Back, and expects the root of the profile. It taps Back again and expects the home to show the same user, with no profile left. Screenshots `embedded-profile` and `closed-home`. +- **Custom page.** The second test taps the `E2E Custom Page` row and expects the text `Rehosted RN body`. It leaves the page with Back on iOS and the device's back button on Android, expects the root of the profile, taps Back, and expects the home. +- **Proof.** `screenshots/profile.png` shows the email in the native profile, and the video shows the home with the same email before the tap. The video of the embedded profile run shows the custom page. All four tests pass on iOS and Android. + +## Gotchas + +- The UserButton is `clerk.userButton.profile` on iOS and the accessibility label `Open user profile` on Android (`nativeUserButton` in `specs/native.ts`). +- Profile rows have no stable identifier on Android, and the iOS rows are inside a sheet, so the specs find them by text (`Manage account`, `Sign out`). +- A seeded user has no name, so the profile header shows only the avatar. The email appears under `Manage account`. +- Open the profile from the UserButton, as a user does. The `embeddedProfile` screen is for what only an inline UserProfileView has: `onHostBack` and custom pages. +- Tap the profile's Back button with `tapProfileBack` from `specs/native.ts`, which taps the middle of its frame. On Android the label `Back` is on a group that holds the button, and `locator.tap()` on it fails with "no parent-owned touch point". +- Do not call `host.screenshot` while a custom page is on screen. On iOS the Back button of the page then does nothing. The video shows the page. +- The test app shows a spinner while the ticket sign-in runs, and `host.launch` waits for the home. diff --git a/integration/expo-native/specs/fixtures.ts b/integration/expo-native/specs/fixtures.ts new file mode 100644 index 00000000000..650e86b25f3 --- /dev/null +++ b/integration/expo-native/specs/fixtures.ts @@ -0,0 +1,136 @@ +import { execFile } from 'node:child_process'; +import { randomBytes } from 'node:crypto'; +import { join } from 'node:path'; +import { test as base } from '@e2e-dev/mobile'; +import { expect, type Locator } from 'e2e'; +import { app as testApp } from './app.ts'; +import { busyWaits, onceTheRunnerIsFree, type BusyWait } from './support/busy-runner.ts'; +import { testUsers } from './support/clerk.ts'; +import { ASSERTION_TIMEOUT_MS, PACKAGE_DIR } from './support/config.ts'; +import { agentDevice, deviceCommand } from './support/device.ts'; +import { fillField } from './support/fill.ts'; +import { readClerk, readRun, readTarget, type Target } from './support/inputs.ts'; +import { appLauncher } from './support/launch.ts'; +import { redact } from './support/secret.ts'; +import { tapOnceUncovered } from './support/tapping.ts'; +import { + APP_ELEMENT_IDS, + CLERK_TEST_CODE, + signedInText, + type AppLocators, + type HostFixture, + type LaunchOptions, + type SeededUser, + type TestEmail, +} from './support/types.ts'; +import { errorScreenElseSavePasswordPrompt, onScreen, signedInAs, signedOut, until, type Sight, type WaitScreen } from './support/waiting.ts'; + +export { expect, CLERK_TEST_CODE }; + +const LAUNCH_TIMEOUT_MS = 60_000; +const POLL_MS = 400; +const TYPING_DELAY_MS = 40; +const CONFIRM_READS = 5; +const ONLY_E2E_WORKER_SLOT = 0; + +const RUN = readRun(process.env); + +const AGENT_DEVICE = join(PACKAGE_DIR, 'node_modules', '.bin', 'agent-device'); + +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +const runnerBusy = (): BusyWait => busyWaits(6, () => sleep(5_000)); + +const randomId = (): string => randomBytes(8).toString('hex'); + +const BUILDS_THIS_WORKER_INSTALLED = new Set(); + +async function adb(target: Target, args: readonly string[]): Promise { + const result = await deviceCommand(target.device, target.platform, args); + if (result.code !== 0) throw new Error(`adb ${args[0]} failed: ${result.stderr.trim() || result.stdout.trim() || `exit ${result.code}`}`); +} + +function typeOnDevice(target: Target, [command, ...operands]: readonly [string, ...string[]]): Promise { + const device = agentDevice(target.device, target.platform); + return new Promise((resolve, reject) => { + const env = { ...process.env, ...device.env }; + execFile(AGENT_DEVICE, [command, '--delay-ms', String(TYPING_DELAY_MS), '--session', `${target.session}-${ONLY_E2E_WORKER_SLOT}`, ...device.selector, '--', ...operands], { env }, (error, _stdout, stderr) => { + if (error === null) resolve(); + else reject(new Error(redact(`agent-device ${command} failed: ${stderr.trim() || error.message}`))); + }); + }); +} + +export const test = base.extend<{ host: HostFixture }>({ + host: async ({ app: e2eApp, device, screen, platform }, use) => { + const target = readTarget(testApp, process.env); + if (target.platform !== platform) throw new Error(`e2e runs the ${platform} target and the inputs name ${target.platform}; pass --target ${target.platform}`); + const clerk = readClerk(process.env); + const users = testUsers(clerk, RUN); + const appId = testApp.id(platform); + const app = Object.fromEntries(Object.entries(APP_ELEMENT_IDS).map(([name, id]) => [name, screen.getByTestId(id)])) as AppLocators; + const launchApp = appLauncher( + { platform, id: appId, entry: testApp.entry(platform, target.build.devServer), buildPath: target.build.path, publishableKey: clerk.publishableKey, run: RUN }, + { + installApp: (path) => device.installApp(path), + signInTicket: users.signInTicket, + openApp: (id, options) => (options === undefined ? device.openApp(id) : device.openApp(id, { relaunch: true, launchArguments: [...options.launchArguments] })), + adb: (args) => adb(target, args), + }, + BUILDS_THIS_WORKER_INSTALLED, + randomId, + ); + + const waitScreen: WaitScreen = { + home: app, + errorScreenElsePrompt: device.locator(errorScreenElseSavePasswordPrompt(APP_ELEMENT_IDS.error)), + dismissPrompt: () => screen.getByRole('button', { name: 'Not Now' }).tap({ timeout: ASSERTION_TIMEOUT_MS }), + }; + const waitFor = (sight: Sight, what: string, timeoutMs: number): Promise => until(sight, what, waitScreen, { timeoutMs, now: Date.now, wait: () => sleep(POLL_MS) }); + + const describeSignedIn = (who: SeededUser | TestEmail): string => + typeof who === 'string' ? `"${signedInText(who)}" with a user ID and a session ID on the home` : `"${signedInText(who.email)}" with user ID ${who.id} and a session ID on the home`; + + function landing(options: LaunchOptions): { readonly sight: Sight; readonly what: string } { + const on = (locator: Locator, what: string) => ({ sight: onScreen(locator, what), what }); + if (options.landsOn !== undefined) return on(options.landsOn, 'the screen the spec names in landsOn'); + const user = options.signedInAs ?? null; + if (user !== null) return { sight: signedInAs(app, user), what: describeSignedIn(user) }; + if (options.keepStorage === true) { + throw new Error('a launch with keepStorage and no signedInAs has no landing the fixture can know; pass landsOn with the locator the launch should show'); + } + return on(app.signIn, 'the "Sign in" button on the home'); + } + + const tap = (control: Locator, busy: BusyWait): Promise => + tapOnceUncovered(() => control.tap({ timeout: ASSERTION_TIMEOUT_MS }), { timeoutMs: ASSERTION_TIMEOUT_MS, now: Date.now, wait: () => sleep(POLL_MS) }, busy); + + const host: HostFixture = { + runId: RUN, + app, + newEmail: async () => users.newEmail(), + newPhone: users.newPhone, + seedUser: users.seed, + async launch(options = {}) { + const lands = landing(options); + const launchId = await launchApp(options); + await waitFor(lands.sight, `${lands.what} after launch ${launchId}`, LAUNCH_TIMEOUT_MS); + }, + expectSignedInAs: (who, timeoutMs = ASSERTION_TIMEOUT_MS) => waitFor(signedInAs(app, who), describeSignedIn(who), timeoutMs), + expectSignedOut: (timeoutMs = ASSERTION_TIMEOUT_MS) => waitFor(signedOut(app), '"Signed out" and no user ID or session ID on the home', timeoutMs), + async screenshot(label) { + await onceTheRunnerIsFree(() => e2eApp.screenshot(label), runnerBusy()); + }, + tap: (control) => tap(control, runnerBusy()), + fill: async (field, text) => + fillField( + field, + typeof text === 'string' ? text : text.use('device-input', (value) => value), + { platform, focused: device.locator('role=textbox focused'), texts: device.locator('role=text'), type: (command) => typeOnDevice(target, command) }, + { tapTimeoutMs: ASSERTION_TIMEOUT_MS, reads: CONFIRM_READS, now: Date.now, wait: () => sleep(POLL_MS) }, + runnerBusy(), + ), + }; + await use(host); + }, +}); diff --git a/integration/expo-native/specs/golden/custom-flow-sign-in/complete.e2e.ts b/integration/expo-native/specs/golden/custom-flow-sign-in/complete.e2e.ts new file mode 100644 index 00000000000..40bc8cb12fb --- /dev/null +++ b/integration/expo-native/specs/golden/custom-flow-sign-in/complete.e2e.ts @@ -0,0 +1,17 @@ +import { test, expect, CLERK_TEST_CODE } from '../../fixtures.ts'; +import { homeLinks } from '../../native.ts'; + +test('useSignIn completes with the email code', async ({ host, screen }) => { + const user = await host.seedUser(); + await host.launch(); + await host.tap(homeLinks(screen).customSignIn); + await host.fill(screen.getByTestId('verify.customSignIn.emailAddress'), user.email); + await host.tap(screen.getByTestId('verify.customSignIn.sendCode')); + await expect(screen.getByTestId('verify.customSignIn.code')).toBeVisible({ timeout: 20_000 }); + await expect(screen.getByTestId('verify.customSignIn.error')).toHaveCount(0); + await host.screenshot('custom-code'); + await host.fill(screen.getByTestId('verify.customSignIn.code'), CLERK_TEST_CODE); + await host.tap(screen.getByTestId('verify.customSignIn.verifyCode')); + await host.expectSignedInAs(user, 30_000); + await host.screenshot('custom-signed-in'); +}); diff --git a/integration/expo-native/specs/golden/custom-flow-sign-up/complete.e2e.ts b/integration/expo-native/specs/golden/custom-flow-sign-up/complete.e2e.ts new file mode 100644 index 00000000000..4a08e720716 --- /dev/null +++ b/integration/expo-native/specs/golden/custom-flow-sign-up/complete.e2e.ts @@ -0,0 +1,18 @@ +import { test, expect, CLERK_TEST_CODE } from '../../fixtures.ts'; +import { homeLinks } from '../../native.ts'; + +test('useSignUp completes with the email code', async ({ host, screen }) => { + const email = await host.newEmail(); + await host.launch(); + await host.tap(homeLinks(screen).customSignUp); + await host.fill(screen.getByTestId('verify.customSignUp.emailAddress'), email); + await host.fill(screen.getByTestId('verify.customSignUp.password'), `Verify-${host.runId}-Pw1!`); + await host.tap(screen.getByTestId('verify.customSignUp.sendCode')); + await expect(screen.getByTestId('verify.customSignUp.code')).toBeVisible({ timeout: 20_000 }); + await expect(screen.getByTestId('verify.customSignUp.error')).toHaveCount(0); + await host.screenshot('custom-signup-code'); + await host.fill(screen.getByTestId('verify.customSignUp.code'), CLERK_TEST_CODE); + await host.tap(screen.getByTestId('verify.customSignUp.verifyCode')); + await host.expectSignedInAs(email, 30_000); + await host.screenshot('custom-signed-up'); +}); diff --git a/integration/expo-native/specs/golden/native-auth-view/complete.e2e.ts b/integration/expo-native/specs/golden/native-auth-view/complete.e2e.ts new file mode 100644 index 00000000000..e40021655e2 --- /dev/null +++ b/integration/expo-native/specs/golden/native-auth-view/complete.e2e.ts @@ -0,0 +1,19 @@ +import { test, expect, CLERK_TEST_CODE } from '../../fixtures.ts'; + +test('signs in through AuthView with the email code', { platforms: ['ios'] }, async ({ host, screen }) => { + const user = await host.seedUser(); + await host.launch({ authMode: 'signIn' }); + await host.tap(host.app.signInFullScreen); + await host.fill(screen.getByTestId('clerk.auth.start.identifier'), user.email); + await host.tap(screen.getByTestId('clerk.auth.start.continue')); + const anotherMethod = screen.getByText('Use another method'); + await expect(anotherMethod).toBeVisible({ timeout: 20_000 }); + await host.tap(anotherMethod); + await host.tap(screen.getByTestId('clerk.auth.signIn.alternativeMethod.email_code')); + await expect(screen.getByTestId('clerk.auth.signIn.code')).toBeVisible({ timeout: 20_000 }); + await expect(screen.getByText(user.email)).toBeVisible(); + await host.screenshot('code-screen'); + await host.fill(screen.getByTestId('clerk.auth.signIn.code'), CLERK_TEST_CODE); + await host.expectSignedInAs(user, 30_000); + await host.screenshot('signed-in'); +}); diff --git a/integration/expo-native/specs/golden/native-auth-view/logo.e2e.ts b/integration/expo-native/specs/golden/native-auth-view/logo.e2e.ts new file mode 100644 index 00000000000..8d591a2fc55 --- /dev/null +++ b/integration/expo-native/specs/golden/native-auth-view/logo.e2e.ts @@ -0,0 +1,28 @@ +import { test, expect } from '../../fixtures.ts'; +import { closeNativeAuth, homeLinks, nativeAuth, signInWithPassword } from '../../native.ts'; + +test('signs in with a password through the AuthView that has a logo, after it closed and reopened', async ({ + host, + screen, + device, + platform, +}) => { + const logo = screen.getByText('E2E Custom Logo'); + const signInWithLogo = homeLinks(screen).authLogo; + const user = await host.seedUser({ password: true }); + await host.launch(); + await host.tap(signInWithLogo); + await expect(logo).toBeVisible({ timeout: 20_000 }); + await expect(nativeAuth(screen, platform).identifier).toBeVisible(); + await host.screenshot('logo'); + await closeNativeAuth(screen, device, platform); + await host.expectSignedOut(15_000); + await expect(logo).toHaveCount(0); + + await host.tap(signInWithLogo); + await expect(logo).toBeVisible({ timeout: 20_000 }); + await host.screenshot('logo-reopened'); + await signInWithPassword(host, screen, platform, user); + await host.expectSignedInAs(user, 45_000); + await host.screenshot('signed-in-with-logo'); +}); diff --git a/integration/expo-native/specs/golden/native-auth-view/opens.e2e.ts b/integration/expo-native/specs/golden/native-auth-view/opens.e2e.ts new file mode 100644 index 00000000000..5ed07149879 --- /dev/null +++ b/integration/expo-native/specs/golden/native-auth-view/opens.e2e.ts @@ -0,0 +1,37 @@ +import { test, expect } from '../../fixtures.ts'; +import { closeNativeAuth, homeLinks, nativeAuth } from '../../native.ts'; + +test('the home full-screen sign-in button shows AuthView with no close button', async ({ host, screen, platform }) => { + await host.launch(); + await host.tap(host.app.signInFullScreen); + await expect(nativeAuth(screen, platform).identifier).toBeVisible({ timeout: 20_000 }); + if (platform === 'ios') await expect(screen.getByTestId('clerk.dismissButton')).toHaveCount(0); + await host.screenshot('auth-full-screen'); +}); + +test('closing the AuthView that the home opened returns to the home', async ({ host, screen, device, platform }) => { + const identifier = nativeAuth(screen, platform).identifier; + await host.launch(); + await host.expectSignedOut(); + await host.tap(host.app.signIn); + await expect(identifier).toBeVisible({ timeout: 20_000 }); + await host.screenshot('auth-from-home'); + await closeNativeAuth(screen, device, platform); + await host.expectSignedOut(15_000); + await expect(host.app.signIn).toBeVisible(); + await expect(identifier).toHaveCount(0); + await host.screenshot('closed-modal'); +}); + +test( + 'the close button of a full-screen AuthView returns to the home', + { platforms: ['ios'] }, + async ({ host, screen }) => { + await host.launch(); + await host.tap(homeLinks(screen).nativeAuth); + await host.tap(screen.getByTestId('clerk.dismissButton').last()); + await host.expectSignedOut(15_000); + await expect(host.app.signIn).toBeVisible(); + await host.screenshot('dismissed-home'); + }, +); diff --git a/integration/expo-native/specs/golden/native-js-sync/sign-in-from-native.e2e.ts b/integration/expo-native/specs/golden/native-js-sync/sign-in-from-native.e2e.ts new file mode 100644 index 00000000000..378db0ef68a --- /dev/null +++ b/integration/expo-native/specs/golden/native-js-sync/sign-in-from-native.e2e.ts @@ -0,0 +1,36 @@ +import { test, expect } from '../../fixtures.ts'; +import { nativeProfile, nativeUserButton, signInWithPassword } from '../../native.ts'; + +test('a native password sign-in reaches the JS hooks and survives a relaunch', async ({ host, screen, platform }) => { + const user = await host.seedUser({ password: true }); + await host.launch(); + await host.tap(host.app.signIn); + await signInWithPassword(host, screen, platform, user); + await host.expectSignedInAs(user, 45_000); + const session = (await host.app.sessionId.textContent()) ?? ''; + await host.screenshot('signed-in'); + + await host.launch({ keepStorage: true, landsOn: host.app.signedIn }); + await host.expectSignedInAs(user); + await expect(host.app.sessionId).toHaveText(session); + await host.screenshot('restored-home'); +}); + +test('a second native sign-in in the same process reaches the JS hooks', async ({ host, screen, platform }) => { + const user = await host.seedUser({ password: true }); + await host.launch(); + await host.tap(host.app.signIn); + await signInWithPassword(host, screen, platform, user); + await host.expectSignedInAs(user, 45_000); + + await host.tap(nativeUserButton(screen, platform)); + const profile = nativeProfile(screen); + await expect(profile.signOut).toBeVisible({ timeout: 30_000 }); + await host.tap(profile.signOut); + await host.expectSignedOut(20_000); + + await host.tap(host.app.signIn); + await signInWithPassword(host, screen, platform, user); + await host.expectSignedInAs(user, 45_000); + await host.screenshot('signed-in-again'); +}); diff --git a/integration/expo-native/specs/golden/native-js-sync/sign-out-from-native.e2e.ts b/integration/expo-native/specs/golden/native-js-sync/sign-out-from-native.e2e.ts new file mode 100644 index 00000000000..2b8f1cf168a --- /dev/null +++ b/integration/expo-native/specs/golden/native-js-sync/sign-out-from-native.e2e.ts @@ -0,0 +1,14 @@ +import { test, expect } from '../../fixtures.ts'; +import { nativeProfile, nativeUserButton } from '../../native.ts'; + +test('signing out in the native profile signs out the JS hooks', async ({ host, screen, platform }) => { + const user = await host.seedUser(); + await host.launch({ signedInAs: user }); + await host.tap(nativeUserButton(screen, platform)); + const profile = nativeProfile(screen); + await expect(profile.signOut).toBeVisible({ timeout: 30_000 }); + await host.screenshot('before-sign-out'); + await host.tap(profile.signOut); + await host.expectSignedOut(20_000); + await host.screenshot('signed-out'); +}); diff --git a/integration/expo-native/specs/golden/native-modules/biometric-availability.e2e.ts b/integration/expo-native/specs/golden/native-modules/biometric-availability.e2e.ts new file mode 100644 index 00000000000..a8af2723009 --- /dev/null +++ b/integration/expo-native/specs/golden/native-modules/biometric-availability.e2e.ts @@ -0,0 +1,18 @@ +import { test, expect } from '../../fixtures.ts'; +import { homeLinks } from '../../native.ts'; + +test('useBiometricCredentials reports what the expo-biometrics native module found on the device', async ({ + host, + screen, + platform, +}) => { + const nativeAnswer = platform === 'ios' ? 'biometric_authentication_unavailable' : 'no_local_credential'; + await host.launch(); + await host.tap(homeLinks(screen).nativeModules); + await host.tap(screen.getByTestId('biometric-availability-button')); + await expect(screen.getByTestId('biometric-availability-result')).toHaveText( + `biometric availability: ${nativeAnswer}`, + { timeout: 15_000 }, + ); + await host.screenshot('biometric-availability'); +}); diff --git a/integration/expo-native/specs/golden/native-modules/biometric-availability.settings.json b/integration/expo-native/specs/golden/native-modules/biometric-availability.settings.json new file mode 100644 index 00000000000..1934771b5b1 --- /dev/null +++ b/integration/expo-native/specs/golden/native-modules/biometric-availability.settings.json @@ -0,0 +1,4 @@ +{ + "config": { "auth_biometric": { "used_for_sign_in": true } }, + "environment": { "auth_config.native_settings.trusted_device_sign_in_enabled": true } +} diff --git a/integration/expo-native/specs/golden/native-modules/google-sign-in.e2e.ts b/integration/expo-native/specs/golden/native-modules/google-sign-in.e2e.ts new file mode 100644 index 00000000000..dd949c8fab7 --- /dev/null +++ b/integration/expo-native/specs/golden/native-modules/google-sign-in.e2e.ts @@ -0,0 +1,33 @@ +import { test, expect } from '../../fixtures.ts'; +import { homeLinks } from '../../native.ts'; + +const GOOGLE_UI_TIMEOUT_MS = 30_000; + +test('useSignInWithGoogle opens the native Google sign-in, and cancelling it on iOS ends the flow with no session', async ({ + host, + screen, + device, + platform, +}) => { + await host.launch(); + await host.tap(homeLinks(screen).nativeModules); + await host.tap(screen.getByTestId('google-sign-in-button')); + if (platform === 'android') { + const googlePage = screen.getByTestId(/^com\.google\.android\.gms:id\//); + let opened = false; + for (const until = Date.now() + GOOGLE_UI_TIMEOUT_MS; !opened && Date.now() < until; ) { + opened = await googlePage.count().then( + count => count > 0, + () => false, + ); + if (!opened) await new Promise(resolve => setTimeout(resolve, 1_000)); + } + expect(opened, 'a page of Google Play services is on screen').toBe(true); + return; + } + await expect(screen.getByText(/accounts\.google\.com/)).toBeVisible({ timeout: GOOGLE_UI_TIMEOUT_MS }); + await host.screenshot('google-system-prompt'); + await device.alert('dismiss'); + await expect(screen.getByTestId('google-result')).toHaveText('Google sign-in was cancelled', { timeout: 15_000 }); + await host.screenshot('google-cancelled'); +}); diff --git a/integration/expo-native/specs/golden/token-cache-persistence/relaunch.e2e.ts b/integration/expo-native/specs/golden/token-cache-persistence/relaunch.e2e.ts new file mode 100644 index 00000000000..018515838c0 --- /dev/null +++ b/integration/expo-native/specs/golden/token-cache-persistence/relaunch.e2e.ts @@ -0,0 +1,30 @@ +import { test, expect } from '../../fixtures.ts'; +import { homeLinks } from '../../native.ts'; + +test('the token cache keeps the session across a relaunch, and a new scope starts signed out', async ({ + host, + screen, +}) => { + const clientToken = screen.getByTestId('verify.tokenCache.clientToken'); + const cachedUser = screen.getByTestId('verify.tokenCache.user'); + const tokenCache = homeLinks(screen).tokenCache; + const user = await host.seedUser(); + await host.launch({ signedInAs: user }); + const session = (await host.app.sessionId.textContent()) ?? ''; + + await host.launch({ keepStorage: true, landsOn: host.app.signedIn }); + await host.expectSignedInAs(user); + await expect(host.app.sessionId).toHaveText(session); + await host.screenshot('restored-home'); + + await host.tap(tokenCache); + await expect(clientToken).toHaveText('client token kept from the last launch: yes', { timeout: 15_000 }); + await expect(cachedUser).toHaveText(`user: ${user.id}`, { timeout: 15_000 }); + await host.screenshot('token-cache'); + + await host.launch(); + await host.tap(tokenCache); + await expect(clientToken).toHaveText('client token kept from the last launch: no', { timeout: 15_000 }); + await expect(cachedUser).toHaveText('user: none', { timeout: 15_000 }); + await host.screenshot('new-scope'); +}); diff --git a/integration/expo-native/specs/golden/user-button-and-profile/embedded-profile.e2e.ts b/integration/expo-native/specs/golden/user-button-and-profile/embedded-profile.e2e.ts new file mode 100644 index 00000000000..4c089cadc11 --- /dev/null +++ b/integration/expo-native/specs/golden/user-button-and-profile/embedded-profile.e2e.ts @@ -0,0 +1,45 @@ +import { test, expect } from '../../fixtures.ts'; +import { embeddedProfile, homeLinks, leaveCustomPage, nativeProfile, tapProfileBack } from '../../native.ts'; + +test('onHostBack closes the embedded profile after native navigation', async ({ host, screen }) => { + const user = await host.seedUser(); + const profile = nativeProfile(screen); + const embedded = embeddedProfile(screen); + await host.launch({ signedInAs: user }); + await host.tap(homeLinks(screen).embeddedProfile); + await expect(profile.manageAccount).toBeVisible({ timeout: 20_000 }); + await host.screenshot('embedded-profile'); + + await host.tap(profile.manageAccount); + await expect(screen.getByText(user.email)).toBeVisible({ timeout: 20_000 }); + await tapProfileBack(screen); + await expect(embedded.customPageRow).toBeVisible({ timeout: 15_000 }); + + await tapProfileBack(screen); + await host.expectSignedInAs(user, 15_000); + await expect(embedded.customPageRow).toHaveCount(0); + await host.screenshot('closed-home'); +}); + +test('a custom page renders React Native content inside the embedded profile', async ({ + host, + screen, + device, + platform, +}) => { + const user = await host.seedUser(); + const profile = nativeProfile(screen); + const embedded = embeddedProfile(screen); + await host.launch({ signedInAs: user }); + await host.tap(homeLinks(screen).embeddedProfile); + await host.tap(embedded.customPageRow); + await expect(embedded.customPageBody).toBeVisible({ timeout: 15_000 }); + + await leaveCustomPage(screen, device, platform); + await expect(profile.manageAccount).toBeVisible({ timeout: 15_000 }); + await expect(embedded.customPageBody).toHaveCount(0); + + await tapProfileBack(screen); + await host.expectSignedInAs(user, 15_000); + await expect(profile.manageAccount).toHaveCount(0); +}); diff --git a/integration/expo-native/specs/golden/user-button-and-profile/profile.e2e.ts b/integration/expo-native/specs/golden/user-button-and-profile/profile.e2e.ts new file mode 100644 index 00000000000..a71f7bff979 --- /dev/null +++ b/integration/expo-native/specs/golden/user-button-and-profile/profile.e2e.ts @@ -0,0 +1,23 @@ +import { test, expect } from '../../fixtures.ts'; +import { nativeProfile, nativeUserButton } from '../../native.ts'; + +test('the home UserButton opens the profile of the signed-in user', async ({ host, screen, platform }) => { + const user = await host.seedUser(); + await host.launch({ signedInAs: user }); + await host.tap(nativeUserButton(screen, platform)); + const profile = nativeProfile(screen); + await expect(profile.manageAccount).toBeVisible({ timeout: 20_000 }); + await host.screenshot('user-button-profile'); + await host.tap(profile.manageAccount); + await expect(screen.getByText(user.email)).toBeVisible({ timeout: 20_000 }); + await host.screenshot('profile'); +}); + +test('the home sign-out button ends the session', async ({ host }) => { + const user = await host.seedUser(); + await host.launch({ signedInAs: user }); + await host.tap(host.app.signOut); + await host.expectSignedOut(); + await expect(host.app.signIn).toBeVisible(); + await host.screenshot('signed-out'); +}); diff --git a/integration/expo-native/specs/native.ts b/integration/expo-native/specs/native.ts new file mode 100644 index 00000000000..0d8ebee4f79 --- /dev/null +++ b/integration/expo-native/specs/native.ts @@ -0,0 +1,137 @@ +import { expect, type Locator } from 'e2e'; +import type { SecretLike, SeededUser } from './support/types.ts'; + +interface Screen { + getByTestId(id: string): Locator; + getByText(text: string | RegExp): Locator; + getByLabel(text: string): Locator; + getByRole(role: 'textbox'): Locator; + getByDisplayValue(value: string): Locator; + tapAt(point: { x: number; y: number }): Promise; +} + +interface Device { + back(): Promise; +} + +interface Host { + readonly app: { readonly signedIn: Locator }; + tap(target: Locator): Promise; + fill(target: Locator, text: string | SecretLike): Promise; +} + +const CLOSE_TAKES_EFFECT_MS = 3_000; +const PASSWORD_SCREEN_TIMEOUT_MS = 20_000; +const SIGNED_IN_TIMEOUT_MS = 45_000; + +export function nativeAuth(screen: Screen, platform: string) { + const ios = platform === 'ios'; + return { + identifier: ios + ? screen.getByTestId('clerk.auth.start.identifier') + : screen.getByText('Enter your email or username'), + continue: ios ? screen.getByTestId('clerk.auth.start.continue') : screen.getByText('Continue'), + passwordLabel: screen.getByText('Enter your password'), + passwordField: screen.getByRole('textbox').last(), + passwordContinue: ios ? screen.getByTestId('clerk.auth.signIn.continue') : screen.getByText('Continue'), + }; +} + +async function gone(locator: Locator, withinMs: number): Promise { + const deadline = Date.now() + withinMs; + while ((await locator.count()) > 0) { + if (Date.now() >= deadline) return false; + await new Promise(resolve => setTimeout(resolve, 300)); + } + return true; +} + +export async function closeNativeAuth(screen: Screen, device: Device, platform: string): Promise { + if (platform !== 'ios') { + await device.back(); + return; + } + const close = screen.getByTestId('clerk.dismissButton').last(); + await close.tap(); + if (!(await gone(close, CLOSE_TAKES_EFFECT_MS))) await close.tap(); +} + +async function reachPasswordScreen(host: Host, screen: Screen, platform: string, user: SeededUser): Promise { + const auth = nativeAuth(screen, platform); + await expect(auth.continue).toBeVisible({ timeout: PASSWORD_SCREEN_TIMEOUT_MS }); + const remembered = screen.getByDisplayValue(user.email); + if ((await remembered.count()) === 0) await host.fill(auth.identifier, user.email); + await host.tap(auth.continue); + await expect(auth.passwordLabel.first()).toBeVisible({ timeout: PASSWORD_SCREEN_TIMEOUT_MS }); + await expect(auth.passwordField).toBeVisible(); + return auth.passwordField; +} + +async function declineAndroidPasswordManager(screen: Screen, signedIn: Locator): Promise { + const prompt = screen.getByText(/Google Password Manager/); + await expect + .poll(async () => (await prompt.count()) > 0 || (await signedIn.count()) > 0, { timeout: SIGNED_IN_TIMEOUT_MS }) + .toBe(true); + if ((await prompt.count()) > 0) { + await screen + .getByText(/^(Not now|Never)$/i) + .first() + .tap(); + } +} + +export async function signInWithPassword( + host: Host, + screen: Screen, + platform: string, + user: SeededUser, +): Promise { + if (user.password === null) throw new Error('seed the user with host.seedUser({ password: true })'); + const field = await reachPasswordScreen(host, screen, platform, user); + await host.fill(field, user.password); + await host.tap(nativeAuth(screen, platform).passwordContinue); + if (platform === 'android') await declineAndroidPasswordManager(screen, host.app.signedIn); +} + +export function homeLinks(screen: Screen) { + return { + nativeAuth: screen.getByTestId('e2e.home.nativeAuth'), + authLogo: screen.getByTestId('e2e.home.authLogo'), + customSignIn: screen.getByTestId('e2e.home.customSignIn'), + customSignUp: screen.getByTestId('e2e.home.customSignUp'), + tokenCache: screen.getByTestId('e2e.home.tokenCache'), + embeddedProfile: screen.getByTestId('e2e.home.embeddedProfile'), + nativeModules: screen.getByTestId('e2e.home.nativeModules'), + }; +} + +export function nativeUserButton(screen: Screen, platform: string): Locator { + return platform === 'ios' ? screen.getByTestId('clerk.userButton.profile') : screen.getByLabel('Open user profile'); +} + +export function nativeProfile(screen: Screen) { + return { + manageAccount: screen.getByText('Manage account'), + signOut: screen.getByText('Sign out'), + }; +} + +export function embeddedProfile(screen: Screen) { + return { + customPageRow: screen.getByText('E2E Custom Page'), + customPageBody: screen.getByText('Rehosted RN body'), + }; +} + +export async function tapProfileBack(screen: Screen): Promise { + const back = screen.getByLabel('Back').last(); + await expect(back).toBeVisible(); + const frame = await back.boundingBox(); + if (frame === null) throw new Error('the Back button of the profile has no frame'); + await screen.tapAt({ x: frame.x + frame.width / 2, y: frame.y + frame.height / 2 }); +} + +export async function leaveCustomPage(screen: Screen, device: Device, platform: string): Promise { + if (platform === 'ios') await tapProfileBack(screen); + else await device.back(); +} diff --git a/integration/expo-native/src/core/MANIFEST b/integration/expo-native/src/core/MANIFEST index fd1a27fb9d8..08a32d5c0b9 100644 --- a/integration/expo-native/src/core/MANIFEST +++ b/integration/expo-native/src/core/MANIFEST @@ -1,4 +1,5 @@ 0ec17d3e3a153410d40bd6cc57ddb259745c77403fa681c6670a3aa6456aa826 e2e.config.ts +f1ac21eeca82c1f04cbb42f0b77d51c798c71842a5a0b97a4660c3265ab601b6 specs/fixtures.ts 38e8e99663059810b616b591dfd79bdcc7781f72b7269875294f3702b3edde84 specs/support/agent.ts 6d1ef7700a89e19d0b1214d5a6815ba757d02857c555b8bf108f2e0c9a64aad3 specs/support/busy-runner.ts b7ecd13380d680a8a0a184cb9ef325baf97804d8b98ea7d676b21ccaf2630f4d specs/support/clerk.ts From 8354a231a4f6a2f543eab73b4c33cc17198c5972 Mon Sep 17 00:00:00 2001 From: Mike Pitre <12040919+mikepitre@users.noreply.github.com> Date: Wed, 7 Oct 2026 20:31:01 -0400 Subject: [PATCH 6/7] test(expo): add the runner's unit tests and run them in CI Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 58 + integration/expo-native/test/agent.test.ts | 237 +++ integration/expo-native/test/android.test.ts | 525 ++++++ .../expo-native/test/app-start.test.ts | 196 ++ integration/expo-native/test/broker.test.ts | 338 ++++ .../expo-native/test/busy-runner.test.ts | 85 + integration/expo-native/test/claims.test.ts | 48 + integration/expo-native/test/clerk.test.ts | 58 + integration/expo-native/test/cli.test.ts | 248 +++ .../expo-native/test/device-command.test.ts | 52 + integration/expo-native/test/down.test.ts | 161 ++ integration/expo-native/test/driver.test.ts | 629 +++++++ integration/expo-native/test/e2e.test.ts | 171 ++ integration/expo-native/test/end-run.test.ts | 33 + integration/expo-native/test/evidence.test.ts | 200 ++ integration/expo-native/test/fill.test.ts | 160 ++ .../expo-native/test/freshness.test.ts | 451 +++++ integration/expo-native/test/git-env.test.ts | 63 + .../expo-native/test/github-report.test.ts | 249 +++ integration/expo-native/test/host.test.ts | 202 ++ integration/expo-native/test/inputs.test.ts | 140 ++ .../expo-native/test/instances.test.ts | 1677 +++++++++++++++++ .../expo-native/test/lease-flow.test.ts | 207 ++ integration/expo-native/test/ledgers.test.ts | 115 ++ integration/expo-native/test/lock.test.ts | 51 + .../expo-native/test/processes.test.ts | 45 + .../expo-native/test/run-groups.test.ts | 669 +++++++ integration/expo-native/test/seam.test.ts | 101 + integration/expo-native/test/secrets.test.ts | 104 + integration/expo-native/test/settings.test.ts | 249 +++ .../expo-native/test/something-ran.test.ts | 47 + integration/expo-native/test/state.test.ts | 45 + integration/expo-native/test/tapping.test.ts | 86 + .../expo-native/test/test-users.test.ts | 171 ++ integration/expo-native/test/typing.test.ts | 318 ++++ integration/expo-native/test/waiting.test.ts | 324 ++++ .../expo-native/testing/sample-inputs.ts | 19 + 37 files changed, 8532 insertions(+) create mode 100644 integration/expo-native/test/agent.test.ts create mode 100644 integration/expo-native/test/android.test.ts create mode 100644 integration/expo-native/test/app-start.test.ts create mode 100644 integration/expo-native/test/broker.test.ts create mode 100644 integration/expo-native/test/busy-runner.test.ts create mode 100644 integration/expo-native/test/claims.test.ts create mode 100644 integration/expo-native/test/clerk.test.ts create mode 100644 integration/expo-native/test/cli.test.ts create mode 100644 integration/expo-native/test/device-command.test.ts create mode 100644 integration/expo-native/test/down.test.ts create mode 100644 integration/expo-native/test/driver.test.ts create mode 100644 integration/expo-native/test/e2e.test.ts create mode 100644 integration/expo-native/test/end-run.test.ts create mode 100644 integration/expo-native/test/evidence.test.ts create mode 100644 integration/expo-native/test/fill.test.ts create mode 100644 integration/expo-native/test/freshness.test.ts create mode 100644 integration/expo-native/test/git-env.test.ts create mode 100644 integration/expo-native/test/github-report.test.ts create mode 100644 integration/expo-native/test/host.test.ts create mode 100644 integration/expo-native/test/inputs.test.ts create mode 100644 integration/expo-native/test/instances.test.ts create mode 100644 integration/expo-native/test/lease-flow.test.ts create mode 100644 integration/expo-native/test/ledgers.test.ts create mode 100644 integration/expo-native/test/lock.test.ts create mode 100644 integration/expo-native/test/processes.test.ts create mode 100644 integration/expo-native/test/run-groups.test.ts create mode 100644 integration/expo-native/test/seam.test.ts create mode 100644 integration/expo-native/test/secrets.test.ts create mode 100644 integration/expo-native/test/settings.test.ts create mode 100644 integration/expo-native/test/something-ran.test.ts create mode 100644 integration/expo-native/test/state.test.ts create mode 100644 integration/expo-native/test/tapping.test.ts create mode 100644 integration/expo-native/test/test-users.test.ts create mode 100644 integration/expo-native/test/typing.test.ts create mode 100644 integration/expo-native/test/waiting.test.ts create mode 100644 integration/expo-native/testing/sample-inputs.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 38fcec32813..a032601372e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -104,6 +104,64 @@ jobs: pnpm changeset status --since=origin/main; fi + expo-native-runner-tests: + needs: [check-permissions] + name: Expo Native Runner Tests + runs-on: 'blacksmith-8vcpu-ubuntu-2204' + defaults: + run: + shell: bash + timeout-minutes: ${{ vars.TIMEOUT_MINUTES_NORMAL && fromJSON(vars.TIMEOUT_MINUTES_NORMAL) || 5 }} + + env: + PACKAGE: integration/expo-native + # Every path the package's unit tests read, so a change that would break one of them runs them. + WATCHED: >- + integration/expo-native + .github/workflows/ci.yml + .github/actionlint.yaml + integration/templates/expo-native + integration/tests/expo-native + packages/expo + packages/expo-biometrics + packages/expo-google-signin + + steps: + - name: Checkout Repo + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + persist-credentials: false + fetch-depth: 2 # The pull request merge commit and its parents, to see what the PR changes + fetch-tags: false + show-progress: false + + - name: Check whether the package changed + id: changed + env: + EVENT: ${{ github.event_name }} + run: | + if [ "$EVENT" != "pull_request" ] || ! git diff --quiet HEAD^1 HEAD -- $WATCHED; then + echo "package=true" >> "$GITHUB_OUTPUT" + fi + + - name: Setup Node + if: steps.changed.outputs.package == 'true' + uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 + with: + node-version: 24.15.0 + + - name: Install the package's own dependencies + if: steps.changed.outputs.package == 'true' + run: npm ci --prefix "$PACKAGE" + + - name: Run the package's unit tests, which include the check of the copied core against its manifest + if: steps.changed.outputs.package == 'true' + run: npm test --prefix "$PACKAGE" + + - name: Typecheck the package + if: steps.changed.outputs.package == 'true' + run: npm run typecheck --prefix "$PACKAGE" + build-packages: needs: [check-permissions] name: Build Packages diff --git a/integration/expo-native/test/agent.test.ts b/integration/expo-native/test/agent.test.ts new file mode 100644 index 00000000000..5db5fdd875e --- /dev/null +++ b/integration/expo-native/test/agent.test.ts @@ -0,0 +1,237 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { execFileSync } from 'node:child_process'; +import { chmodSync, mkdirSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { isAbsolute, join, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { describe, it } from 'node:test'; +import { inspect } from 'node:util'; +import { agentCheck, agentEnvironment, takeAgent } from '../src/core/agent.ts'; +import { BACKUP_AGENT_MODEL, DEFAULT_AGENT_MODEL, UnusableAgentKey, readAgent } from '../specs/support/agent.ts'; +import { composeE2EConfig } from '../specs/support/config.ts'; +import { invokeE2E, planE2E } from '../src/core/e2e.ts'; +import { sealEvidence } from '../src/core/evidence.ts'; +import { withoutClerkKeys } from '../src/core/keys.ts'; +import { redact, usedSecretValues } from '../specs/support/secret.ts'; +import type { EvidencePath, EvidenceRecord } from '../src/core/types.ts'; +import { IOS_APP, SAMPLE_INPUTS } from '../testing/sample-inputs.ts'; + +const GATEWAY_KEY = `gw_${'k'.repeat(20)}UnitTestOnly${'z'.repeat(20)}`; +const TARGET = SAMPLE_INPUTS.target; + +function keyFile(mode: number, text = `${GATEWAY_KEY}\n`): string { + const file = join(mkdtempSync(join(tmpdir(), 'verify-agent-key-')), 'ai-gateway-key'); + writeFileSync(file, text, { mode }); + chmodSync(file, mode); + return file; +} + +function modelOf(config: ReturnType): { readonly provider: string; readonly modelId: string } { + return config.agents?.default?.model as unknown as { readonly provider: string; readonly modelId: string }; +} + +function fakeE2E(): string { + const packageDir = mkdtempSync(join(tmpdir(), 'verify-agent-')); + const bin = join(packageDir, 'node_modules', '.bin'); + mkdirSync(bin, { recursive: true }); + const script = [ + '#!/usr/bin/env node', + "const seen = (name) => process.env[name] ?? 'unset';", + "console.log(`key=${seen('AI_GATEWAY_API_KEY')}`);", + "console.log(`file=${seen('AI_GATEWAY_API_KEY_FILE')}`);", + "console.log(`platform=${seen('CLERK_PLATFORM_API_KEY')}`);", + "console.log(`secret=${seen('CLERK_SECRET_KEY')}`);", + "console.log(`app=${seen('CLERK_E2E_APP_PATH')}`);", + "console.log(`device=${seen('CLERK_E2E_DEVICE')}`);", + "console.log(`argv=${process.argv.slice(2).join(' ')}`);", + ].join('\n'); + writeFileSync(join(bin, 'e2e'), `${script}\n`, { mode: 0o755 }); + return packageDir; +} + +async function invoke(agent: Parameters[4]): Promise<{ readonly lines: readonly string[]; readonly log: string }> { + const packageDir = fakeE2E(); + const log = join(packageDir, 'e2e.log') as EvidencePath; + const lines: string[] = []; + const result = await invokeE2E({ args: ['run', 'specs/explored/probe.e2e.ts'], env: { CLERK_E2E_DEVICE: 'nowhere' } }, log, packageDir, (line) => lines.push(line), agent); + assert.equal(result.exitCode, 0); + return { lines, log: readFileSync(log, 'utf8') }; +} + +describe('no agent without a gateway key', () => { + it('composes a config with no agent', () => { + const config = composeE2EConfig(IOS_APP, TARGET, {}); + assert.deepEqual(Object.keys(config).sort(), ['assertionTimeout', 'targets', 'tests', 'timeout', 'trace', 'workers']); + assert.equal(config.agents, undefined); + assert.equal(composeE2EConfig(IOS_APP, TARGET, { AI_GATEWAY_API_KEY: ' ' }).agents, undefined, 'a blank key is no key'); + }); + + it('gives the e2e process no gateway variable, also when the caller has one in its own environment', async () => { + const before = { key: process.env.AI_GATEWAY_API_KEY, file: process.env.AI_GATEWAY_API_KEY_FILE }; + process.env.AI_GATEWAY_API_KEY = GATEWAY_KEY; + process.env.AI_GATEWAY_API_KEY_FILE = '/home/x/ai-gateway-key'; + try { + const { lines } = await invoke(null); + assert.deepEqual(lines.slice(0, 2), ['key=unset', 'file=unset']); + } finally { + for (const [name, value] of [['AI_GATEWAY_API_KEY', before.key], ['AI_GATEWAY_API_KEY_FILE', before.file]] as const) { + if (value === undefined) delete process.env[name]; + else process.env[name] = value; + } + } + }); + + it('needs no ai package, so a worktree that has not installed it still runs every spec without an agent step', () => { + const script = [ + "import { registerHooks } from 'node:module';", + "registerHooks({ resolve: (specifier, context, next) => { if (specifier === 'ai') throw new Error('ai is not installed'); return next(specifier, context); } });", + `const { composeE2EConfig } = await import(${JSON.stringify(fileURLToPath(new URL('../specs/support/config.ts', import.meta.url)))});`, + `const target = ${JSON.stringify(TARGET)};`, + "const app = { platforms: ['ios'], id: () => 'com.clerk.sample', entry: () => ({ kind: 'binary' }) };", + "console.log(Object.keys(composeE2EConfig(app, target, {})).sort().join(','));", + "try { composeE2EConfig(app, target, { AI_GATEWAY_API_KEY: 'x'.repeat(24) }); console.log('composed an agent'); } catch (error) { console.log(error.message); }", + ].join('\n'); + const printed = execFileSync(process.execPath, ['--input-type=module', '-e', script], { encoding: 'utf8' }).trim().split('\n'); + assert.deepEqual(printed, ['assertionTimeout,targets,tests,timeout,trace,workers', 'ai is not installed']); + }); + + it('tells doctor that no agent is configured, as a passing check', () => { + assert.equal(takeAgent({})(), null); + assert.deepEqual(agentCheck(takeAgent({})), { + id: 'agent', + ok: true, + detail: 'none: AI_GATEWAY_API_KEY and AI_GATEWAY_API_KEY_FILE are not set, so agent.act and agent.assert have no model', + }); + }); +}); + +describe('the agent with a gateway key', () => { + it('is the default model on the Vercel AI Gateway, with the replay cache off', () => { + assert.equal(DEFAULT_AGENT_MODEL, 'anthropic/claude-haiku-5.5'); + const config = composeE2EConfig(IOS_APP, TARGET, { AI_GATEWAY_API_KEY: GATEWAY_KEY }); + assert.deepEqual(Object.keys(config.agents ?? {}), ['default']); + assert.deepEqual({ provider: modelOf(config).provider, modelId: modelOf(config).modelId }, { provider: 'gateway', modelId: 'anthropic/claude-haiku-5.5' }); + assert.equal(config.cache, 'off', 'every agent step is judged by the model on the screen of this run'); + }); + + it('names one backup model that the gateway tries when the default model fails', () => { + assert.equal(BACKUP_AGENT_MODEL, 'openai/gpt-6-luna-fast'); + const config = composeE2EConfig(IOS_APP, TARGET, { AI_GATEWAY_API_KEY: GATEWAY_KEY }); + assert.deepEqual(config.agents?.default?.providerOptions, { gateway: { models: ['openai/gpt-6-luna-fast'] } }); + assert.equal(readAgent({ AI_GATEWAY_API_KEY: GATEWAY_KEY })?.backup, 'openai/gpt-6-luna-fast'); + }); + + it('reads the key from a file that only the user can read', () => { + const file = keyFile(0o600); + assert.equal(modelOf(composeE2EConfig(IOS_APP, TARGET, { AI_GATEWAY_API_KEY_FILE: file })).modelId, 'anthropic/claude-haiku-5.5'); + assert.equal(readAgent({ AI_GATEWAY_API_KEY_FILE: file })?.credential.variable, 'AI_GATEWAY_API_KEY_FILE'); + assert.equal(readAgent({ AI_GATEWAY_API_KEY: GATEWAY_KEY, AI_GATEWAY_API_KEY_FILE: file })?.credential.variable, 'AI_GATEWAY_API_KEY', 'the inline key wins, as it does for the Platform API key'); + }); + + it('hands e2e the absolute path of a key file named relative to the caller, because e2e runs in the package directory', () => { + const file = keyFile(0o600); + const named = relative(process.cwd(), file); + assert.equal(isAbsolute(named), false); + assert.deepEqual(agentEnvironment(readAgent({ AI_GATEWAY_API_KEY_FILE: named })), { AI_GATEWAY_API_KEY_FILE: file }); + }); + + it('refuses a key file that others can read, a missing one, and an empty one', () => { + const open = keyFile(0o644); + assert.throws(() => readAgent({ AI_GATEWAY_API_KEY_FILE: open }), (error: UnusableAgentKey) => error instanceof UnusableAgentKey && error.fix === `chmod 600 ${open}`); + assert.throws(() => readAgent({ AI_GATEWAY_API_KEY_FILE: '/nowhere/ai-gateway-key' }), (error: UnusableAgentKey) => error instanceof UnusableAgentKey && /does not exist/.test(error.what)); + const empty = keyFile(0o600, '\n'); + assert.throws(() => readAgent({ AI_GATEWAY_API_KEY_FILE: empty }), (error: UnusableAgentKey) => error instanceof UnusableAgentKey && /holds no key/.test(error.what)); + const directory = mkdtempSync(join(tmpdir(), 'verify-agent-key-dir-')); + chmodSync(directory, 0o700); + assert.throws(() => readAgent({ AI_GATEWAY_API_KEY_FILE: directory }), (error: UnusableAgentKey) => error instanceof UnusableAgentKey && /cannot be read \(EISDIR\)/.test(error.what)); + assert.equal(agentCheck(takeAgent({ AI_GATEWAY_API_KEY_FILE: directory })).ok, false, 'doctor reports it and does not crash'); + const check = agentCheck(takeAgent({ AI_GATEWAY_API_KEY_FILE: open })); + assert.deepEqual({ id: check.id, ok: check.ok, fix: check.fix }, { id: 'agent', ok: false, fix: `chmod 600 ${open}` }); + }); + + it('tells doctor the model and where the key came from', () => { + assert.equal(agentCheck(takeAgent({ AI_GATEWAY_API_KEY: GATEWAY_KEY })).detail, 'anthropic/claude-haiku-5.5, with openai/gpt-6-luna-fast as its backup, through the Vercel AI Gateway; key from AI_GATEWAY_API_KEY'); + const file = keyFile(0o600); + assert.equal(agentCheck(takeAgent({ AI_GATEWAY_API_KEY_FILE: file })).detail, 'anthropic/claude-haiku-5.5, with openai/gpt-6-luna-fast as its backup, through the Vercel AI Gateway; key from AI_GATEWAY_API_KEY_FILE'); + }); +}); + +describe('the gateway key stays out of what a run writes', () => { + it('leaves the environment that every other child process inherits', () => { + const file = keyFile(0o600); + const env: NodeJS.ProcessEnv = { AI_GATEWAY_API_KEY: GATEWAY_KEY, AI_GATEWAY_API_KEY_FILE: file, HOME: '/home/x' }; + const agent = takeAgent(env); + assert.deepEqual(env, { HOME: '/home/x' }); + assert.equal(agent()?.model, DEFAULT_AGENT_MODEL, 'the agent was read before the variables were removed'); + assert.deepEqual(withoutClerkKeys({ AI_GATEWAY_API_KEY: GATEWAY_KEY, AI_GATEWAY_API_KEY_FILE: file, HOME: '/home/x' }), { HOME: '/home/x' }); + }); + + it('prints as a placeholder and is known to the log redaction and the evidence scan', () => { + const agent = readAgent({ AI_GATEWAY_API_KEY: GATEWAY_KEY }); + assert.ok(agent !== null); + for (const printed of [String(agent.credential.key), JSON.stringify(agent), inspect(agent, { depth: 8 }), JSON.stringify(agentCheck(() => agent))]) { + assert.equal(printed.includes(GATEWAY_KEY), false, printed); + } + assert.equal(redact(`Authorization: Bearer ${GATEWAY_KEY}`), 'Authorization: Bearer '); + assert.ok(usedSecretValues().includes(GATEWAY_KEY), 'the evidence scan looks for it'); + + const file = keyFile(0o600, ` ${GATEWAY_KEY}-from-file\n`); + takeAgent({ AI_GATEWAY_API_KEY_FILE: file })(); + assert.ok(usedSecretValues().includes(`${GATEWAY_KEY}-from-file`), 'a key from a file is known once it is read'); + }); + + it('is not in the composed config where a report or a log could print it', () => { + const config = composeE2EConfig(IOS_APP, TARGET, { AI_GATEWAY_API_KEY: GATEWAY_KEY }); + assert.equal(JSON.stringify(config).includes(GATEWAY_KEY), false); + assert.equal(inspect(config, { depth: 10, showHidden: true }).includes(GATEWAY_KEY), false); + }); + + it('reaches the e2e process through its environment only, and the log of that process is redacted', async () => { + const inline = readAgent({ AI_GATEWAY_API_KEY: GATEWAY_KEY }); + const plan = planE2E( + SAMPLE_INPUTS, + [{ kind: 'explored', path: 'specs/explored/probe.e2e.ts', feature: null }], + { verb: 'run', selection: { selectors: ['specs/explored/probe.e2e.ts'] }, video: false, retries: 0, githubReport: false, waitSeconds: 0 }, + '/package', + '/package/.verify/runs/r20261007-120000-abcd/e2e' as EvidencePath, + ); + assert.equal(JSON.stringify(plan).includes('GATEWAY'), false, 'the planned command line and its environment name no gateway variable'); + + const handed = await invoke(inline); + assert.deepEqual(handed.lines, ['key=', 'file=unset', 'platform=unset', 'secret=unset', 'app=unset', 'device=nowhere', 'argv=run specs/explored/probe.e2e.ts']); + assert.equal(handed.log.includes(GATEWAY_KEY), false); + assert.match(handed.log, /^key=$/m); + + const file = keyFile(0o600); + const fromFile = readAgent({ AI_GATEWAY_API_KEY_FILE: file }); + assert.deepEqual(agentEnvironment(fromFile), { AI_GATEWAY_API_KEY_FILE: file }, 'a key in a file is handed over as the path, so no environment holds its value'); + assert.deepEqual((await invoke(fromFile)).lines.slice(0, 2), ['key=unset', `file=${file}`]); + }); + + it('taints a run whose files hold it', () => { + readAgent({ AI_GATEWAY_API_KEY: GATEWAY_KEY }); + const dir = mkdtempSync(join(tmpdir(), 'verify-agent-run-')) as EvidencePath; + writeFileSync(join(dir, 'e2e.log'), `model call failed: 401 for key ${GATEWAY_KEY}\n`); + writeFileSync(join(dir, 'app.log'), 'nothing here\n'); + const sealed = sealEvidence(dir, { run: 'r20261007-120000-abcd' } as unknown as Omit); + assert.deepEqual(sealed.tainted, [join(dir, 'e2e.log')]); + }); +}); + +describe('the environment the CLI gives the e2e process', () => { + it("holds no secret key of the instance and no setting from the caller's shell, only the settings the CLI names", async () => { + const exported = { CLERK_SECRET_KEY: 'sk_test_exportedByTheCaller', CLERK_E2E_APP_PATH: '/somewhere/else/E2EHost.app', CLERK_E2E_DEVICE: 'a-device-the-caller-exported' }; + const before = Object.fromEntries(Object.keys(exported).map((name) => [name, process.env[name]])); + Object.assign(process.env, exported); + try { + const { lines } = await invoke(null); + assert.deepEqual(lines.slice(3, 6), ['secret=unset', 'app=unset', 'device=nowhere']); + } finally { + for (const [name, value] of Object.entries(before)) { + if (value === undefined) delete process.env[name]; + else process.env[name] = value; + } + } + }); +}); diff --git a/integration/expo-native/test/android.test.ts b/integration/expo-native/test/android.test.ts new file mode 100644 index 00000000000..4113cffe700 --- /dev/null +++ b/integration/expo-native/test/android.test.ts @@ -0,0 +1,525 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { spawn } from 'node:child_process'; +import { once } from 'node:events'; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { isRunning, run } from '../src/core/exec.ts'; +import type { EvidencePath, LocalLease } from '../src/core/types.ts'; +import { readClaim, takeSlot } from '../src/core/claims.ts'; +import { LOG_FILTER, RECORD_SIZE, emulatorArgs, laneSettingsCommand, laneSettingsHold, laneSettingsReadCommand, logFilter, logcatSince } from '../src/platform/android/emulator.ts'; +import { lanePort, localAndroidBackend, laneSerial, parseAdbDevices, startScreenrecord, terminateGroup } from '../src/platform/android/local.ts'; +import { ensureLaneAvd, jdkCheck, localAvailability, resolveJavaHome, sdkRoot, systemImage, type Machine } from '../src/platform/android/sdk.ts'; + +function machineHomedInScratch(dir: string, overrides: Partial = {}): Machine { + return { os: 'darwin', arch: 'arm64', home: dir, env: {}, kvm: join(dir, 'kvm'), ...overrides }; +} + +const doctorOptions = { live: false, worktree: '/nowhere', progress: () => undefined }; + +function fakeJdk(root: string, version: string): string { + const home = join(root, `jdk-${version}`); + mkdirSync(home, { recursive: true }); + writeFileSync(join(home, 'release'), `IMPLEMENTOR="test"\nJAVA_VERSION="${version}"\n`); + return home; +} + +describe('android lanes', () => { + it('puts slot 1 on emulator-5560 and slot 2 on emulator-5562', () => { + assert.equal(lanePort(1), 5560); + assert.equal(laneSerial(2), 'emulator-5562'); + }); + + it('reads adb devices, skipping the header and blank lines', () => { + const devices = parseAdbDevices('List of devices attached\nemulator-5560\tdevice\nemulator-5562\toffline\n\n'); + assert.deepEqual([...devices], [['emulator-5560', 'device'], ['emulator-5562', 'offline']]); + }); + + it('asks logcat for lines since the run started, in epoch seconds', () => { + assert.equal(logcatSince(new Date(1_791_014_000_123)), '1791014000.123'); + }); + + it('keeps the host, SDK, network, and React Native console tags and silences the rest', () => { + assert.deepEqual(LOG_FILTER.slice(-1), ['*:S']); + for (const tag of ['ClerkVerify:V', 'ClerkLog:V', 'OkHttp:V', 'ReactNativeJS:V']) assert.ok(LOG_FILTER.includes(tag), tag); + }); + + it('adds a host log predicate before the final silence spec', () => { + assert.deepEqual(logFilter('Expo:V ReactNative:W').slice(-3), ['Expo:V', 'ReactNative:W', '*:S']); + assert.deepEqual(logFilter(), LOG_FILTER); + }); + + it('boots with no -prop, which the emulator refuses outside qemu.* and never shows in getprop, so the lane is marked after boot', () => { + for (const os of ['darwin', 'linux'] as const) assert.equal(emulatorArgs(5560, os).includes('-prop'), false, os); + }); + + it('boots a headless lane with software graphics on Linux and leaves the Mac lane as it was', () => { + assert.deepEqual(emulatorArgs(5560, 'darwin'), ['-avd', 'Clerk_Verify_Pixel', '-read-only', '-no-window', '-no-audio', '-no-boot-anim', '-port', '5560']); + assert.deepEqual(emulatorArgs(5560, 'linux'), ['-avd', 'Clerk_Verify_Pixel', '-read-only', '-no-window', '-no-audio', '-no-boot-anim', '-gpu', 'swiftshader_indirect', '-port', '5560']); + }); + + it('turns off the window, transition, and animator animations on a lane, and reads the same settings back', () => { + for (const scale of ['window_animation_scale', 'transition_animation_scale', 'animator_duration_scale']) { + assert.ok(laneSettingsCommand().split(' && ').includes(`settings put global ${scale} 0`), scale); + assert.ok(laneSettingsReadCommand().split(' && ').includes(`settings get global ${scale}`), scale); + } + assert.equal(laneSettingsCommand().split(' && ').length, laneSettingsReadCommand().split(' && ').length); + }); + + it('hides the system\'s "isn\'t responding" and "keeps stopping" dialogs on a lane, which cover the app and which agent-device will not tap through', () => { + assert.ok(laneSettingsCommand().split(' && ').includes('settings put global hide_error_dialogs 1')); + assert.ok(laneSettingsReadCommand().split(' && ').includes('settings get global hide_error_dialogs')); + }); + + it('counts a lane as set up only when every setting reads back as written', () => { + const written = laneSettingsCommand().split(' && ').map((command) => command.split(' ').at(-1) as string); + assert.equal(laneSettingsHold(written.join('\n')), true); + assert.equal(laneSettingsHold(`${written.map((value) => `${value}.0`).join('\n')}\n`), true, 'the settings provider may print 0 as 0.0'); + assert.equal(laneSettingsHold(['1.0', ...written.slice(1)].join('\n')), false, 'one animation scale still on'); + assert.equal(laneSettingsHold([...written.slice(0, -1), '0'].join('\n')), false, 'error dialogs still shown'); + assert.equal(laneSettingsHold(written.slice(1).join('\n')), false, 'a setting that did not answer'); + assert.equal(laneSettingsHold(written.map(() => 'null').join('\n')), false, 'a device that never had them set'); + assert.equal(laneSettingsHold(''), false); + }); + + it('records at the panel\'s aspect ratio, because the codec refuses 1280x2856 and screenrecord then falls back to 720x1280', () => { + const [width, height] = RECORD_SIZE.split('x').map(Number) as [number, number]; + assert.ok(Math.abs(width / height - 1280 / 2856) < 0.001, RECORD_SIZE); + }); + +}); + +describe('android JDK', () => { + const root = mkdtempSync(join(tmpdir(), 'verify-jdk-')); + const jbr = fakeJdk(root, '21.0.8'); + const java17 = fakeJdk(root, '17.0.17'); + + it('fails JAVA_HOME on Java 17 with a fix that names the Java 21 JBR', () => { + const check = jdkCheck({ JAVA_HOME: java17 }, jbr); + assert.equal(check.ok, false); + assert.match(check.detail, /Java 17/); + assert.equal(check.fix, `export JAVA_HOME="${jbr}"`); + }); + + it('builds with the JBR when JAVA_HOME is unset, and with JAVA_HOME when it is 21', () => { + assert.deepEqual(resolveJavaHome({}, jbr), { ok: true, home: jbr, detail: 'Java 21 from the Android Studio JBR' }); + const java21 = fakeJdk(root, '21.0.2'); + const chosen = resolveJavaHome({ JAVA_HOME: java21 }, join(root, 'missing')); + assert.equal(chosen.ok && chosen.home, java21); + }); + + it('fails with an install fix when there is no JBR and no JAVA_HOME', () => { + const check = jdkCheck({}, join(root, 'missing')); + assert.equal(check.ok, false); + assert.match(check.fix ?? '', /install Android Studio/); + }); +}); + +describe('android screenrecord', () => { + it('stops on the device, waits for pidof to empty, and only then pulls, because a recording stopped from the host has no moov atom', async () => { + const dir = mkdtempSync(join(tmpdir(), 'verify-adb-')); + const log = join(dir, 'adb.log'); + const adb = join(dir, 'adb'); + writeFileSync( + adb, + [ + '#!/bin/bash', + `echo "$*" >> ${log}`, + `state=${dir}/recording`, + 'case "$*" in', + ' *"shell screenrecord"*) touch "$state"; while [ -f "$state" ]; do sleep 0.1; done ;;', + ' *"shell pidof screenrecord"*) [ -f "$state" ] && echo 4242 ;;', + ' *"shell pkill -INT screenrecord"*) rm -f "$state" ;;', + ' *" pull "*) echo mp4 > "${@: -1}" ;;', + 'esac', + '', + ].join('\n'), + ); + chmodSync(adb, 0o755); + const into = join(dir, 'r20261003-040814-846e') as EvidencePath; + mkdirSync(into); + + const recording = await startScreenrecord({ adbBin: adb, serial: 'emulator-5560', into }); + const video = await recording.stop(); + + assert.equal(video, join(into, 'video.mp4')); + assert.ok(existsSync(video)); + const calls = readFileSync(log, 'utf8').trim().split('\n'); + const record = calls.findIndex((c) => c.includes('shell screenrecord')); + const kill = calls.findIndex((c) => c.includes('pkill -INT screenrecord')); + const pull = calls.findIndex((c) => c.includes(' pull ')); + const lastPidof = calls.findLastIndex((c) => c.includes('pidof screenrecord')); + assert.match(calls[record]!, new RegExp(`--size ${RECORD_SIZE} --time-limit 0 /data/local/tmp/verify-r20261003-040814-846e\\.mp4`)); + assert.ok(record < kill && kill < lastPidof && lastPidof < pull, calls.join('\n')); + assert.ok(calls.some((c) => c.includes('rm -f /data/local/tmp/verify-r20261003-040814-846e.mp4')), 'removes the device copy'); + }); + + it('fails stop with adb\'s error when the pull fails', async () => { + const dir = mkdtempSync(join(tmpdir(), 'verify-adb-')); + const adb = fakeTool(dir, 'adb', [ + `state=${dir}/recording`, + 'case "$*" in', + ' *"shell screenrecord"*) touch "$state"; while [ -f "$state" ]; do sleep 0.1; done ;;', + ' *"shell pidof screenrecord"*) [ -f "$state" ] && echo 4242 ;;', + ' *"shell pkill -INT screenrecord"*) rm -f "$state" ;;', + ' *" pull "*) echo "adb: error: remote object does not exist" >&2; exit 1 ;;', + 'esac', + ]); + const into = join(dir, 'r20261003-040814-846e') as EvidencePath; + mkdirSync(into); + const recording = await startScreenrecord({ adbBin: adb, serial: 'emulator-5560', into }); + await assert.rejects(recording.stop(), { code: 'NOT_READY', message: /remote object does not exist/ }); + }); +}); + +function fakeTool(dir: string, name: string, body: readonly string[]): string { + const path = join(dir, name); + writeFileSync(path, ['#!/bin/bash', `echo "$*" >> ${join(dir, 'calls.log')}`, ...body, ''].join('\n')); + chmodSync(path, 0o755); + return path; +} + +describe('android lane ownership', () => { + function setup(avd: string, laneProperty: (ownNonce: string) => string) { + const dir = mkdtempSync(join(tmpdir(), 'verify-lane-')); + const claimsDir = join(dir, 'claims'); + const claim = takeSlot(claimsDir, 'android', 1, 0, join(dir, 'worktree'))!; + const killed = join(dir, 'killed'); + const adbBin = fakeTool(dir, 'adb', [ + 'case "$*" in', + ` "devices") echo "List of devices attached"; [ -f ${killed} ] || printf "emulator-5560\\tdevice\\n" ;;`, + ` *"emu avd name"*) printf "${avd}\\nOK\\n" ;;`, + ` *"getprop debug.verify.lane"*) echo "${laneProperty(claim.nonce)}" ;;`, + ' *"getprop sys.boot_completed"*) echo 1 ;;', + ` *"emu kill"*) touch ${killed} ;;`, + 'esac', + ]); + const emulatorBin = fakeTool(dir, 'emulator', ['echo Clerk_Verify_Pixel']); + const lease: LocalLease = { backend: 'local', platform: 'android', slot: 1, deviceName: 'verify-android-1', deviceId: 'emulator-5560', claimNonce: claim.nonce, acquiredAt: '', installedBuild: null }; + const calls = () => readFileSync(join(dir, 'calls.log'), 'utf8'); + return { backend: localAndroidBackend({ claimsDir, adbBin, emulatorBin, machine: machineHomedInScratch(dir) }), lease, dir, calls }; + } + + it('never kills another AVD that sits on a lane port, and frees the claim', async () => { + const { backend, lease, calls } = setup('Pixel_9_Pro', () => ''); + await backend.release(lease); + assert.doesNotMatch(calls(), /emu kill/); + assert.equal(await backend.check(lease), 'lost'); + }); + + it('never kills a Clerk_Verify_Pixel emulator booted for another claim', async () => { + const { backend, lease, calls } = setup('Clerk_Verify_Pixel', () => 'someone-elses-claim'); + await backend.release(lease); + assert.doesNotMatch(calls(), /emu kill/); + }); + + it('kills the lane it booted', async () => { + const { backend, lease, calls } = setup('Clerk_Verify_Pixel', (own) => own); + assert.equal(await backend.check(lease), 'held'); + await backend.release(lease); + assert.match(calls(), /-s emulator-5560 emu kill/); + }); + + it('doctor flags a foreign emulator on a lane port with a kill command for its owner', async () => { + const { backend } = setup('Pixel_9_Pro', () => ''); + const lanePorts = (await backend.doctorChecks(doctorOptions)).device.find((c) => c.id === 'lane-ports'); + assert.equal(lanePorts?.ok, false); + assert.match(lanePorts?.detail ?? '', /emulator-5560 \(Pixel_9_Pro, not a verify lane\)/); + assert.match(lanePorts?.fix ?? '', /^adb -s emulator-5560 emu kill, but only if that emulator is yours/); + }); + + it('doctor passes a lane port that holds this claim\'s own lane', async () => { + const { backend } = setup('Clerk_Verify_Pixel', (own) => own); + assert.equal((await backend.doctorChecks(doctorOptions)).device.find((c) => c.id === 'lane-ports')?.ok, true); + }); + + it('reports a foreign emulator on a lane port in POOL_FULL instead of claiming it', async () => { + const { backend, lease, dir } = setup('Pixel_9_Pro', () => ''); + await backend.release(lease); + takeSlot(join(dir, 'claims'), 'android', 2, 0, join(dir, 'other')); + await assert.rejects(backend.acquire({ platform: 'android', worktree: join(dir, 'worktree'), waitSeconds: 0, app: null as never, retryWith: 'e2e-tests/bin/control-clerk-android up --wait ', progress: () => undefined }), { + code: 'POOL_FULL', + message: /emulator-5560 \(Pixel_9_Pro, not a verify lane\), verify-android-2 \(held by .*other\)/, + fix: /`adb -s emulator-5560 emu kill` frees a lane, but only if that emulator is yours/, + }); + }); +}); + +describe('android lane settings at boot', () => { + function bootingLane(settingsReadBack: string) { + const dir = mkdtempSync(join(tmpdir(), 'verify-lane-settings-')); + const claimsDir = join(dir, 'claims'); + const booted = join(dir, 'booted'); + const killed = join(dir, 'killed'); + const mark = join(dir, 'mark'); + const emulatorPid = join(dir, 'emulator.pid'); + const adbBin = fakeTool(dir, 'adb', [ + 'case "$*" in', + ` "devices") echo "List of devices attached"; if [ -f ${booted} ] && [ ! -f ${killed} ]; then printf "emulator-5560\\tdevice\\n"; fi ;;`, + ' *"emu avd name"*) printf "Clerk_Verify_Pixel\\nOK\\n" ;;', + ` *"emu kill"*) touch ${killed}; kill "$(cat ${emulatorPid})" ;;`, + ` *"setprop debug.verify.lane "*) echo "\${@: -1}" | sed 's/.* //' > ${mark} ;;`, + ` *"getprop debug.verify.lane"*) cat ${mark} ;;`, + ' *"getprop sys.boot_completed"*) echo 1 ;;', + ' *"getprop init.svc.bootanim"*) echo stopped ;;', + ' *"getprop persist.sys.locale"*) echo en-US ;;', + ` *"settings get global"*) printf "${settingsReadBack}" ;;`, + 'esac', + ]); + const emulatorBin = fakeTool(dir, 'emulator', ['case "$*" in', ' *-list-avds*) echo Clerk_Verify_Pixel ;;', ` *) echo $$ > ${emulatorPid}; touch ${booted}; exec sleep 20 ;;`, 'esac']); + const backend = localAndroidBackend({ claimsDir, adbBin, emulatorBin, emulatorsDir: join(dir, 'emulators'), machine: machineHomedInScratch(dir) }); + const request = { platform: 'android' as const, worktree: join(dir, 'worktree'), waitSeconds: 0, app: null as never, retryWith: 'up --wait ', progress: () => undefined }; + return { backend, request, claimsDir, killed, calls: () => readFileSync(join(dir, 'calls.log'), 'utf8') }; + } + + it('writes the lane settings after boot and leases the lane when they read back as written', async () => { + const written = laneSettingsCommand().split(' && ').map((command) => command.split(' ').at(-1) as string); + const { backend, request, calls } = bootingLane(`${written.join('\\n')}\\n`); + const lease = await backend.acquire(request); + assert.equal(lease.deviceId, 'emulator-5560'); + assert.ok(calls().includes(`-s emulator-5560 shell ${laneSettingsCommand()}`), 'the settings were written on the lane'); + await backend.release(lease); + }); + + it('fails up when a lane setting did not take, kills the lane it booted, and frees the slot', async () => { + const { backend, request, claimsDir, killed } = bootingLane('0\\n0\\n0\\nnull\\n'); + await assert.rejects(backend.acquire(request), { code: 'NOT_READY', message: /emulator-5560 did not take the lane settings: .*hide_error_dialogs read back as 0, 0, 0, null/ }); + assert.equal(existsSync(killed), true, 'the emulator this claim booted was killed'); + assert.equal(readClaim(claimsDir, 'android', 1).claim, null, 'slot 1 is free again'); + }); +}); + +describe('android lanes verify spawned', () => { + function fakeEmulatorProcess(port: number): { readonly pid: number; readonly startedAt: number } { + const child = spawn('bash', ['-c', `exec -a "qemu-system-aarch64-headless -avd Clerk_Verify_Pixel -read-only -port ${port}" sleep 60`], { detached: true, stdio: 'ignore' }); + child.unref(); + return { pid: child.pid!, startedAt: Date.now() }; + } + + async function setup(pidRecord: (own: string) => object | null) { + const dir = mkdtempSync(join(tmpdir(), 'verify-spawned-')); + const emulatorsDir = join(dir, 'emulators'); + mkdirSync(emulatorsDir); + const claimsDir = join(dir, 'claims'); + const worktree = join(dir, 'worktree'); + const claim = takeSlot(claimsDir, 'android', 1, 0, worktree)!; + const adbBin = fakeTool(dir, 'adb', ['case "$*" in', ' "devices") echo "List of devices attached" ;;', 'esac']); + const emulatorBin = fakeTool(dir, 'emulator', ['echo Clerk_Verify_Pixel']); + const emulator = fakeEmulatorProcess(5560); + await new Promise((resolve) => setTimeout(resolve, 300)); + const record = pidRecord(claim.nonce); + if (record !== null) writeFileSync(join(emulatorsDir, 'android-1.pid'), JSON.stringify({ ...emulator, ...record })); + const backend = localAndroidBackend({ claimsDir, adbBin, emulatorBin, emulatorsDir, machine: machineHomedInScratch(dir) }); + return { backend, emulator, emulatorsDir, worktree }; + } + + it('reclaims an interrupted boot that left its pid file, through the next up\'s reap', async () => { + const { backend, emulator, emulatorsDir, worktree } = await setup((own) => ({ nonce: own })); + assert.ok(isRunning(emulator)); + const [stale] = await backend.reapable(worktree); + await backend.release(stale!); + assert.equal(isRunning(emulator), false, 'the emulator verify spawned is gone'); + assert.equal(existsSync(join(emulatorsDir, 'android-1.pid')), false, 'the pid file is removed'); + }); + + it('never kills a hand-booted Clerk_Verify_Pixel with no pid file', async () => { + const { backend, emulator, worktree } = await setup(() => null); + const [stale] = await backend.reapable(worktree); + await backend.release(stale!); + assert.ok(isRunning(emulator)); + process.kill(-emulator.pid, 'SIGKILL'); + }); + + it('never kills a Clerk_Verify_Pixel whose pid file names another claim', async () => { + const { backend, emulator, worktree } = await setup(() => ({ nonce: 'another-claim' })); + const [stale] = await backend.reapable(worktree); + await backend.release(stale!); + assert.ok(isRunning(emulator)); + process.kill(-emulator.pid, 'SIGKILL'); + }); + + it('doctor does not flag a claimed lane that is still booting, before its marker is set', async () => { + const dir = mkdtempSync(join(tmpdir(), 'verify-booting-')); + const emulatorsDir = join(dir, 'emulators'); + mkdirSync(emulatorsDir); + const claim = takeSlot(join(dir, 'claims'), 'android', 1, 0, join(dir, 'worktree'))!; + const adbBin = fakeTool(dir, 'adb', [ + 'case "$*" in', + ' "devices") printf "List of devices attached\\nemulator-5560\\toffline\\n" ;;', + ' *"emu avd name"*) printf "Clerk_Verify_Pixel\\nOK\\n" ;;', + 'esac', + ]); + const emulator = fakeEmulatorProcess(5560); + await new Promise((resolve) => setTimeout(resolve, 300)); + writeFileSync(join(emulatorsDir, 'android-1.pid'), JSON.stringify({ ...emulator, nonce: claim.nonce })); + const backend = localAndroidBackend({ claimsDir: join(dir, 'claims'), adbBin, emulatorBin: fakeTool(dir, 'emulator', ['echo Clerk_Verify_Pixel']), emulatorsDir, machine: machineHomedInScratch(dir) }); + assert.equal((await backend.doctorChecks(doctorOptions)).device.find((c) => c.id === 'lane-ports')?.ok, true); + process.kill(-emulator.pid, 'SIGKILL'); + }); + + it('stops an emulator that has already exited without throwing, both before Node has reaped it, when macOS answers the kill with EPERM, and after, when the answer is ESRCH', async () => { + const child = spawn('sh', ['-c', 'exit 0'], { detached: true, stdio: 'ignore' }); + const exited = once(child, 'exit'); + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 500); + assert.equal(child.exitCode, null, 'Node has not yet seen the exit'); + assert.doesNotThrow(() => terminateGroup(child.pid!)); + await exited; + assert.doesNotThrow(() => terminateGroup(child.pid!)); + assert.throws(() => terminateGroup(Number.NaN), { code: 'ERR_INVALID_ARG_TYPE' }, 'a failure that is not about a process that is gone still surfaces'); + }); + + it('never kills a reused pid that is no longer the emulator it recorded', async () => { + const { backend, emulator, worktree } = await setup((own) => ({ nonce: own, startedAt: Date.now() - 600_000 })); + const [stale] = await backend.reapable(worktree); + await backend.release(stale!); + assert.ok(isRunning(emulator)); + process.kill(-emulator.pid, 'SIGKILL'); + }); +}); + +describe('whether this machine can run the emulator', () => { + function fakeSdk(dir: string, machine: Machine, options: { readonly image?: boolean } = {}): string { + const root = join(dir, 'sdk'); + for (const tool of [join('emulator', 'emulator'), join('platform-tools', 'adb')]) { + mkdirSync(join(root, tool, '..'), { recursive: true }); + writeFileSync(join(root, tool), ''); + } + if (options.image !== false) { + const image = join(root, ...systemImage(machine).split(';')); + mkdirSync(image, { recursive: true }); + writeFileSync(join(image, 'system.img'), ''); + } + return root; + } + const scratch = () => mkdtempSync(join(tmpdir(), 'verify-machine-')); + + it('says yes on a Mac that has the SDK and the system image, lane AVD or not', () => { + const dir = scratch(); + const mac = machineHomedInScratch(dir); + const root = fakeSdk(dir, mac); + const found = localAvailability({ ...mac, env: { ANDROID_HOME: root } }); + assert.deepEqual(found, { usable: true, why: `this Mac runs the emulator itself, from the SDK at ${root}` }); + }); + + it('finds an SDK in the OS default place and through the tools on PATH', () => { + const dir = scratch(); + const linux = machineHomedInScratch(dir, { os: 'linux', arch: 'x64' }); + const root = fakeSdk(dir, linux); + assert.equal(sdkRoot({ ...linux, env: { PATH: `/usr/bin:${join(root, 'platform-tools')}` } }), root); + const home = scratch(); + mkdirSync(join(home, 'Android'), { recursive: true }); + assert.equal(sdkRoot(machineHomedInScratch(home, { os: 'linux' })), join(home, 'Android', 'Sdk')); + assert.equal(sdkRoot(machineHomedInScratch(home, { os: 'darwin' })), join(home, 'Library', 'Android', 'sdk')); + }); + + it('says no and names what is missing when there is no SDK or no system image', () => { + const dir = scratch(); + const none = localAvailability(machineHomedInScratch(dir)); + assert.equal(none.usable, false); + assert.match(none.why, /^no Android SDK with an emulator and adb \(looked in .*Library\/Android\/sdk\)$/); + assert.match(none.fix ?? '', /ANDROID_HOME/); + + const linux = machineHomedInScratch(dir, { os: 'linux', arch: 'x64' }); + writeFileSync(linux.kvm, ''); + const root = fakeSdk(dir, linux, { image: false }); + const noImage = localAvailability({ ...linux, env: { ANDROID_HOME: root } }); + assert.equal(noImage.why, `the SDK at ${root} has no system image system-images;android-36;google_apis;x86_64`); + assert.match(noImage.fix ?? '', /^sdkmanager "system-images;android-36;google_apis;x86_64"/); + }); + + it('on Linux needs a /dev/kvm this user can open for reading and writing', () => { + const dir = scratch(); + const linux = machineHomedInScratch(dir, { os: 'linux', arch: 'x64' }); + const withSdk = { ...linux, env: { ANDROID_HOME: fakeSdk(dir, linux) } }; + + const missing = { usable: false, why: `there is no ${linux.kvm}, so this machine has no hardware virtualization for the emulator` }; + assert.deepEqual(localAvailability(withSdk), missing); + assert.deepEqual(localAvailability(linux), missing, 'with no SDK either, it names the one thing no install fixes, and offers no fix'); + + writeFileSync(linux.kvm, ''); + chmodSync(linux.kvm, 0o000); + const rootOpensAnyFile = process.getuid?.() === 0; + if (!rootOpensAnyFile) { + const closed = localAvailability(withSdk); + assert.equal(closed.usable, false); + assert.match(closed.why, /^this user cannot open .*kvm for reading and writing/); + assert.match(closed.fix ?? '', /udev/); + } + + chmodSync(linux.kvm, 0o666); + const open = localAvailability(withSdk); + assert.equal(open.usable, true); + assert.match(open.why, /kvm opens for reading and writing and the SDK at .* has the system image$/); + }); + + it('says no on an OS the lanes do not run on', () => { + assert.deepEqual(localAvailability(machineHomedInScratch(scratch(), { os: 'win32' })), { usable: false, why: 'the lane emulator runs on macOS and Linux, and this machine runs win32' }); + }); + + it('writes the lane AVD once, on this machine\'s system image, and never touches one that exists', () => { + const dir = scratch(); + const linux = machineHomedInScratch(dir, { os: 'linux', arch: 'x64' }); + assert.equal(ensureLaneAvd(linux), 'created'); + const config = readFileSync(join(dir, '.android', 'avd', 'Clerk_Verify_Pixel.avd', 'config.ini'), 'utf8'); + for (const line of ['image.sysdir.1=system-images/android-36/google_apis/x86_64/', 'hw.lcd.width=1280', 'hw.lcd.height=2856', 'hw.lcd.density=480', 'abi.type=x86_64']) assert.ok(config.includes(`${line}\n`), line); + assert.match(readFileSync(join(dir, '.android', 'avd', 'Clerk_Verify_Pixel.ini'), 'utf8'), new RegExp(`^path=${join(dir, '.android', 'avd', 'Clerk_Verify_Pixel.avd')}$`, 'm')); + + writeFileSync(join(dir, '.android', 'avd', 'Clerk_Verify_Pixel.avd', 'config.ini'), 'image.sysdir.1=system-images/android-36/google_apis/arm64-v8a/\nhand=made\n'); + assert.equal(ensureLaneAvd(linux), 'exists'); + assert.match(readFileSync(join(dir, '.android', 'avd', 'Clerk_Verify_Pixel.avd', 'config.ini'), 'utf8'), /hand=made/); + }); + + it('finds an AVD that lives where its pointer file says, and leaves both files alone', () => { + const dir = scratch(); + const mac = machineHomedInScratch(dir); + const root = fakeSdk(dir, mac); + const elsewhere = join(dir, 'other-volume', 'Clerk_Verify_Pixel.avd'); + mkdirSync(elsewhere, { recursive: true }); + mkdirSync(join(dir, '.android', 'avd'), { recursive: true }); + const pointer = `avd.ini.encoding=UTF-8\npath=${elsewhere}\ntarget=android-35\n`; + writeFileSync(join(dir, '.android', 'avd', 'Clerk_Verify_Pixel.ini'), pointer); + writeFileSync(join(elsewhere, 'config.ini'), 'image.sysdir.1=system-images/android-35/google_apis/arm64-v8a/\n'); + assert.equal(ensureLaneAvd(mac), 'exists'); + assert.equal(readFileSync(join(dir, '.android', 'avd', 'Clerk_Verify_Pixel.ini'), 'utf8'), pointer); + assert.equal(existsSync(join(dir, '.android', 'avd', 'Clerk_Verify_Pixel.avd')), false); + assert.match(localAvailability({ ...mac, env: { ANDROID_HOME: root } }).why, /has no system image system-images;android-35;google_apis;arm64-v8a, which the Clerk_Verify_Pixel AVD names$/); + }); + + it('uses the ARM image on a Mac whose Node reports an Intel CPU, as it does under Rosetta', () => { + const dir = scratch(); + const root = fakeSdk(dir, machineHomedInScratch(dir)); + const rosetta = machineHomedInScratch(dir, { arch: 'x64', env: { ANDROID_HOME: root } }); + assert.equal(systemImage(rosetta), 'system-images;android-36;google_apis;arm64-v8a'); + assert.equal(localAvailability(rosetta).usable, true); + ensureLaneAvd(rosetta); + assert.match(readFileSync(join(dir, '.android', 'avd', 'Clerk_Verify_Pixel.avd', 'config.ini'), 'utf8'), /^abi\.type=arm64-v8a$/m); + }); + + it('checks the system image an existing lane AVD names, not the default one', () => { + const dir = scratch(); + const mac = machineHomedInScratch(dir); + const root = fakeSdk(dir, mac); + mkdirSync(join(dir, '.android', 'avd', 'Clerk_Verify_Pixel.avd'), { recursive: true }); + writeFileSync(join(dir, '.android', 'avd', 'Clerk_Verify_Pixel.ini'), ''); + writeFileSync(join(dir, '.android', 'avd', 'Clerk_Verify_Pixel.avd', 'config.ini'), 'image.sysdir.1=system-images/android-35/google_apis_playstore/arm64-v8a/\n'); + const found = localAvailability({ ...mac, env: { ANDROID_HOME: root } }); + assert.equal(found.usable, false); + assert.match(found.why, /has no system image system-images;android-35;google_apis_playstore;arm64-v8a, which the Clerk_Verify_Pixel AVD names$/); + assert.match(found.fix ?? '', /^sdkmanager "system-images;android-35;google_apis_playstore;arm64-v8a"/, 'the fix installs the image that AVD names'); + }); + + it('looks for the AVD where the emulator does: ANDROID_AVD_HOME, then ANDROID_USER_HOME, and an empty value is unset', () => { + const dir = scratch(); + const written = (env: Record) => { + const home = mkdtempSync(join(dir, 'home-')); + ensureLaneAvd(machineHomedInScratch(home, { env })); + return { home, at: (root: string) => existsSync(join(root, 'Clerk_Verify_Pixel.ini')) }; + }; + assert.ok(written({ ANDROID_AVD_HOME: join(dir, 'avds') }).at(join(dir, 'avds'))); + assert.ok(written({ ANDROID_USER_HOME: join(dir, 'user') }).at(join(dir, 'user', 'avd'))); + const unset = written({ ANDROID_AVD_HOME: '' }); + assert.ok(unset.at(join(unset.home, '.android', 'avd'))); + }); +}); diff --git a/integration/expo-native/test/app-start.test.ts b/integration/expo-native/test/app-start.test.ts new file mode 100644 index 00000000000..44bd1145453 --- /dev/null +++ b/integration/expo-native/test/app-start.test.ts @@ -0,0 +1,196 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { describe, it } from 'node:test'; +import { newTestEmail } from '../specs/support/clerk.ts'; +import { appLauncher, appStart, performAppStart, type LaunchedApp } from '../specs/support/launch.ts'; +import { Secret } from '../specs/support/secret.ts'; +import type { AppEntry, SeededUser } from '../src/core/types.ts'; +import { SAMPLE_PUBLISHABLE_KEY, SAMPLE_RUN } from '../testing/sample-inputs.ts'; + +const devClient: AppEntry = { + kind: 'dev-client', + launchArguments: ['--ez', 'EXDevMenuIsOnboardingFinished', 'true'], + openLink: 'exp+app://expo-development-client/?url=http%3A%2F%2F10.0.2.2%3A8082', + androidActivity: '.MainActivity', +}; + +describe('appStart', () => { + it('opens a binary host with only the launch arguments', () => { + assert.deepEqual(appStart('ios', 'com.clerk.E2EHost', { kind: 'binary' }, ['-verifyAuthMode', 'signUp']), { kind: 'open-app', launchArguments: ['-verifyAuthMode', 'signUp'] }); + }); + + it('prepends the dev-client arguments on iOS', () => { + const entry: AppEntry = { ...devClient, launchArguments: ['-EXDevMenuIsOnboardingFinished', 'YES'], openLink: null }; + assert.deepEqual(appStart('ios', 'com.clerk.expo', entry, ['-verifyAuthMode', 'signUp']), { + kind: 'open-app', + launchArguments: ['-EXDevMenuIsOnboardingFinished', 'YES', '-verifyAuthMode', 'signUp'], + }); + }); + + it('force-stops and starts the activity with am on Android, where the launcher intent agent-device sends crashes expo-dev-launcher, quoting every token for the device shell', () => { + const start = appStart('android', 'com.clerk.expo', devClient, ['--es', 'verifyInitialIdentifier', "it's me"]); + assert.deepEqual(start, { + kind: 'adb', + commands: [ + ['shell', "am force-stop 'com.clerk.expo'"], + [ + 'shell', + "'am' 'start' '-W' '-n' 'com.clerk.expo/.MainActivity' '-d' 'exp+app://expo-development-client/?url=http%3A%2F%2F10.0.2.2%3A8082' '--ez' 'EXDevMenuIsOnboardingFinished' 'true' '--es' 'verifyInitialIdentifier' 'it'\\''s me'", + ], + ], + }); + }); + + it('refuses an iOS dev client with an openLink, which it could not pass', () => { + assert.throws(() => appStart('ios', 'com.clerk.expo', devClient, []), /cannot use openLink yet/); + }); + + it('refuses an Android dev client with no activity', () => { + assert.throws(() => appStart('android', 'com.clerk.expo', { ...devClient, androidActivity: null }, []), /needs androidActivity/); + }); + + it('after the adb commands, opens the app with no options so the agent-device session binds without a relaunch', async () => { + const calls: string[] = []; + const driver = { + openApp: async (appId: string, options?: { readonly relaunch: true; readonly launchArguments: readonly string[] }) => + void calls.push(options === undefined ? `openApp ${appId}` : `openApp ${appId} relaunch ${options.launchArguments.join(' ')}`), + adb: async (args: readonly string[]) => void calls.push(`adb ${args.join(' ')}`), + }; + await performAppStart(appStart('android', 'com.clerk.expo', devClient, []), 'com.clerk.expo', driver); + assert.equal(calls.length, 3); + assert.match(calls[0]!, /^adb shell am force-stop/); + assert.match(calls[1]!, /^adb shell 'am' 'start'/); + assert.equal(calls[2], 'openApp com.clerk.expo'); + calls.length = 0; + await performAppStart(appStart('ios', 'com.clerk.E2EHost', { kind: 'binary' }, ['-verifyAuthMode', 'signUp']), 'com.clerk.E2EHost', driver); + assert.deepEqual(calls, ['openApp com.clerk.E2EHost relaunch -verifyAuthMode signUp']); + }); + + it('opens the app once more when the device did not open it the first time, and fails when the second try fails too', async () => { + const failing = (failures: number): { readonly opens: () => number; readonly driver: Parameters[2] } => { + let opens = 0; + return { + opens: () => opens, + driver: { + openApp: async () => { + opens += 1; + if (opens <= failures) throw new Error('open com.clerk.E2EHost failed: xcrun timed out after 15000ms'); + }, + adb: async () => {}, + }, + }; + }; + const once = failing(1); + await performAppStart(appStart('ios', 'com.clerk.E2EHost', { kind: 'binary' }, []), 'com.clerk.E2EHost', once.driver); + assert.equal(once.opens(), 2); + const twice = failing(2); + await assert.rejects(performAppStart(appStart('ios', 'com.clerk.E2EHost', { kind: 'binary' }, []), 'com.clerk.E2EHost', twice.driver), /xcrun timed out/); + assert.equal(twice.opens(), 2); + const android = failing(1); + await performAppStart(appStart('android', 'com.clerk.expo', devClient, []), 'com.clerk.expo', android.driver); + assert.equal(android.opens(), 2); + }); + + it('reports the errors of both tries when the second open fails too, the first being the one the device gave', async () => { + const errors = ['open com.clerk.E2EHost failed: xcrun timed out after 15000ms', 'device.openApp exceeded its timeout of 30000ms']; + let opens = 0; + const driver = { openApp: async () => Promise.reject(new Error(errors[(opens += 1) - 1])), adb: async () => {} }; + await assert.rejects(performAppStart(appStart('ios', 'com.clerk.E2EHost', { kind: 'binary' }, []), 'com.clerk.E2EHost', driver), (error: Error) => { + assert.equal(error.message, `the app did not open in two tries. The first failed with: ${errors[0]}. The second failed with: ${errors[1]}`); + assert.equal((error.cause as Error).message, errors[1]); + return true; + }); + assert.equal(opens, 2); + }); + + it('keeps a sign-in ticket that a failed open echoes out of the error it reports', async () => { + const ticket = new Secret('ticket', 'ticket_that_an_error_echoes'); + const echo = ticket.use('launch-argument', (plain) => `simctl launch -verifySignInTicket ${plain} failed`); + const driver = { openApp: async () => Promise.reject(new Error(echo)), adb: async () => {} }; + await assert.rejects(performAppStart(appStart('ios', 'com.clerk.E2EHost', { kind: 'binary' }, []), 'com.clerk.E2EHost', driver), (error: Error) => { + assert.equal(error.message, 'the app did not open in two tries. The first failed with: simctl launch -verifySignInTicket failed. The second failed with: simctl launch -verifySignInTicket failed'); + return true; + }); + }); +}); + +describe('a launch the host fixture starts', () => { + const E2E_HOST: LaunchedApp = { platform: 'ios', id: 'com.clerk.E2EHost', entry: { kind: 'binary' }, buildPath: null, publishableKey: SAMPLE_PUBLISHABLE_KEY, run: SAMPLE_RUN }; + const user: SeededUser = { id: 'user_1', email: newTestEmail(SAMPLE_RUN), phone: null, password: null }; + + function device(app: LaunchedApp = E2E_HOST, installedBuilds = new Set()) { + const calls: string[] = []; + let ids = 0; + const launch = appLauncher( + app, + { + installApp: async (path) => void calls.push(`install ${path}`), + signInTicket: async (who) => (calls.push(`ticket for ${who.id}`), new Secret('ticket', `ticket_of_${who.id}`)), + openApp: async (appId, options) => void calls.push(options === undefined ? `open ${appId}` : `open ${appId} ${options.launchArguments.join(' ')}`), + adb: async (args) => void calls.push(`adb ${args.join(' ')}`), + }, + installedBuilds, + () => `id${(ids += 1)}`, + ); + return { calls, launch }; + } + + const always = `-verifyPublishableKey ${SAMPLE_PUBLISHABLE_KEY} -verifyRunId ${SAMPLE_RUN}`; + + it('relaunches the app with the instance, the run, a new storage scope, and a launch id that it returns', async () => { + const { calls, launch } = device(); + assert.equal(await launch({}), 'id2'); + assert.equal(await launch({}), 'id4'); + assert.deepEqual(calls, [`open com.clerk.E2EHost ${always} -verifyStorageScope id1 -verifyLaunchId id2`, `open com.clerk.E2EHost ${always} -verifyStorageScope id3 -verifyLaunchId id4`]); + }); + + it('hands the app the mode, the identifier, and the log level a launch names', async () => { + const { calls, launch } = device(); + await launch({ authMode: 'signUp', initialIdentifier: 'someone+clerk_test@example.com', debugLogs: true }); + assert.deepEqual(calls, [`open com.clerk.E2EHost ${always} -verifyStorageScope id1 -verifyLaunchId id2 -verifyAuthMode signUp -verifyInitialIdentifier someone+clerk_test@example.com -verifyLogLevel debug`]); + }); + + it('keeps the storage of the launch before it only when the launch asks to', async () => { + const { calls, launch } = device(); + await launch({}); + await launch({ keepStorage: true }); + await launch({}); + assert.deepEqual(calls.map((call) => /-verifyStorageScope (\S+)/.exec(call)?.[1]), ['id1', 'id1', 'id4']); + const first = device(); + await first.launch({ keepStorage: true }); + assert.match(first.calls[0]!, /-verifyStorageScope id1 /, 'a first launch has no storage to keep'); + }); + + it('mints a sign-in ticket for the user a launch starts signed in as, and hands it to the app', async () => { + const { calls, launch } = device(); + await launch({ signedInAs: user }); + assert.deepEqual(calls, ['ticket for user_1', `open com.clerk.E2EHost ${always} -verifyStorageScope id1 -verifyLaunchId id2 -verifySignInTicket ticket_of_user_1`]); + }); + + it('installs the build the settings name before the first launch of a worker, and not again', async () => { + const installedBuilds = new Set(); + const named = { ...E2E_HOST, buildPath: '/builds/E2EHost.app' }; + const firstTest = device(named, installedBuilds); + await firstTest.launch({}); + await firstTest.launch({}); + assert.deepEqual(firstTest.calls.map((call) => call.split(' ')[0]), ['install', 'open', 'open']); + assert.equal(firstTest.calls[0], 'install /builds/E2EHost.app'); + const nextTest = device(named, installedBuilds); + await nextTest.launch({}); + assert.deepEqual(nextTest.calls.map((call) => call.split(' ')[0]), ['open']); + const afterAFailure = new Set(); + const failing = appLauncher(named, { installApp: async () => Promise.reject(new Error('install failed')), signInTicket: async () => assert.fail('no ticket'), openApp: async () => assert.fail('opened'), adb: async () => {} }, afterAFailure, () => 'id'); + await assert.rejects(failing({}), /install failed/); + const retry = device(named, afterAFailure); + await retry.launch({}); + assert.deepEqual(retry.calls.map((call) => call.split(' ')[0]), ['install', 'open'], 'a build that failed to install is installed at the next launch'); + }); + + it('starts a dev client through adb on Android with the same launch inputs', async () => { + const { calls, launch } = device({ ...E2E_HOST, platform: 'android', id: 'com.clerk.expo', entry: devClient }); + await launch({ authMode: 'signIn' }); + assert.equal(calls.length, 3); + assert.match(calls[1]!, /^adb shell 'am' 'start' .*'--es' 'verifyStorageScope' 'id1' '--es' 'verifyLaunchId' 'id2' '--es' 'verifyAuthMode' 'signIn'$/); + assert.equal(calls[2], 'open com.clerk.expo'); + }); +}); diff --git a/integration/expo-native/test/broker.test.ts b/integration/expo-native/test/broker.test.ts new file mode 100644 index 00000000000..13e3407c80a --- /dev/null +++ b/integration/expo-native/test/broker.test.ts @@ -0,0 +1,338 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { mkdtempSync, readFileSync, statSync } from 'node:fs'; +import net from 'node:net'; +import { networkInterfaces, tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { TEST_PHONES, newTestEmail, runEmailPrefix, testUsers } from '../specs/support/clerk.ts'; +import { Secret, usedSecretValues } from '../specs/support/secret.ts'; +import { startBroker, type Broker } from '../src/core/broker.ts'; +import { REPLACE_AT_USERS } from '../src/core/clerk.ts'; +import type { PublishableKey, RunId } from '../src/core/types.ts'; +import { openWorkspace } from '../src/core/workspace.ts'; + +const SECRET_KEY = 'sk_test_standInUnitTest0123456789'; +const keys = { pk: 'pk_test_ZXhhbXBsZS5jbGVyay5hY2NvdW50cy5kZXYk' as PublishableKey, sk: new Secret('clerk-secret-key', SECRET_KEY) }; + +interface Forwarded { + readonly method: string; + readonly url: string; + readonly authorization: string | null; + readonly body: Record | null; + readonly text: string | null; +} + +type Clerk = (request: Forwarded) => { readonly status: number; readonly body: unknown }; + +let minted = 0; + +const emptyClerk: Clerk = (request) => { + if (request.method === 'GET') return { status: 200, body: [] }; + if (request.url.endsWith('/sign_in_tokens')) return { status: 200, body: { object: 'sign_in_token', token: `ticket_${(minted += 1)}_for_${String(request.body?.user_id)}_0123456789` } }; + return { status: 200, body: { object: 'user', id: 'user_created_1', email_addresses: [{ email_address: 'private@example.com' }] } }; +}; + +async function standIn(clerk: Clerk = emptyClerk) { + const dir = mkdtempSync(join(tmpdir(), 'verify-stand-in-')); + const workspace = openWorkspace({ packageDir: dir, worktree: dir, home: join(dir, 'home') }); + const { run, scratch } = workspace.newRun(); + const forwarded: Forwarded[] = []; + const bapi = (async (url: string | URL, init?: RequestInit) => { + const request: Forwarded = { + method: init?.method ?? 'GET', + url: String(url), + authorization: new Headers(init?.headers).get('authorization'), + body: typeof init?.body === 'string' ? (JSON.parse(init.body) as Record) : null, + text: typeof init?.body === 'string' ? init.body : null, + }; + forwarded.push(request); + const answer = clerk(request); + return new Response(JSON.stringify(answer.body), { status: answer.status }); + }) as typeof fetch; + const broker = await startBroker(run, workspace, scratch, { keys: () => keys, fetch: bapi }); + const ask = async (method: string, path: string, body?: unknown, token: string = readFileSync(broker.tokenFile, 'utf8')) => { + const response = await fetch(`${broker.url}${path}`, { method, headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' }, ...(body === undefined ? {} : { body: typeof body === 'string' ? body : JSON.stringify(body) }) }); + const text = await response.text(); + return { status: response.status, json: (text === '' ? null : JSON.parse(text)) as { errors?: { code: string; long_message: string }[] } & Record }; + }; + const users = testUsers({ publishableKey: keys.pk, access: { kind: 'stand-in', url: broker.url, tokenFile: broker.tokenFile } }, run); + return { broker, workspace, run, forwarded, ask, users }; +} + +const stopping = async (broker: Broker, body: () => Promise): Promise => { + try { + return await body(); + } finally { + await broker.stop(); + } +}; + +const whyRefused = (answer: { readonly json: { errors?: { long_message: string }[] } }): string => answer.json.errors?.[0]?.long_message ?? ''; + +describe("the CLI's stand-in for the Backend API", () => { + it('forwards the three calls the tests make with the instance key, which the test process never holds', async () => { + const s = await standIn(); + await stopping(s.broker, async () => { + const user = await s.users.seed({ password: true }); + const ticket = await s.users.signInTicket(user); + await s.users.newPhone(); + assert.deepEqual( + s.forwarded.map((request) => [request.method, request.url.split('?')[0], request.authorization]), + [ + ['POST', 'https://api.clerk.com/v1/users', `Bearer ${SECRET_KEY}`], + ['POST', 'https://api.clerk.com/v1/sign_in_tokens', `Bearer ${SECRET_KEY}`], + ['GET', 'https://api.clerk.com/v1/users', `Bearer ${SECRET_KEY}`], + ], + ); + assert.equal(readFileSync(s.broker.tokenFile, 'utf8').includes(SECRET_KEY), false, 'the token the test process reads is not the key'); + assert.deepEqual(s.forwarded[1]!.body, { user_id: 'user_created_1', expires_in_seconds: 120 }); + assert.equal(user.email.startsWith(runEmailPrefix(s.run)), true); + assert.deepEqual( + s.workspace.entries().flatMap((entry) => (entry.kind === 'user' ? [[entry.run, entry.userId, entry.email]] : [])), + [[s.run, 'user_created_1', user.email]], + 'the run ledgers each user it created', + ); + assert.match(ticket.use('launch-argument', (plain) => plain), /^ticket_\d+_for_user_created_1_/); + }); + }); + + it('learns the password and the ticket that pass through it, so the CLI redacts them and scans the evidence for them', async () => { + const s = await standIn(); + await stopping(s.broker, async () => { + const password = 'a-password-only-the-test-process-made-Aa1!'; + const created = await s.ask('POST', '/users', { email_address: [newTestEmail(s.run)], password, bypass_client_trust: true }); + assert.equal(usedSecretValues().includes(password), true); + const ticket = await s.ask('POST', '/sign_in_tokens', { user_id: created.json.id, expires_in_seconds: 120 }); + assert.equal(typeof ticket.json.token, 'string'); + assert.equal(usedSecretValues().includes(String(ticket.json.token)), true); + }); + }); + + it('learns the pieces a test types the password in, and its own token', async () => { + const s = await standIn(); + await stopping(s.broker, async () => { + const password = 'a-password-typed-in-pieces-of-sixteen-Aa1!'; + await s.ask('POST', '/users', { email_address: [newTestEmail(s.run)], password, bypass_client_trust: true }); + for (const piece of ['a-password-typed', '-in-pieces-of-si', 'xteen-Aa1!']) assert.equal(usedSecretValues().includes(piece), true, piece); + assert.equal(usedSecretValues().includes(readFileSync(s.broker.tokenFile, 'utf8')), true); + }); + }); + + it('refuses a request without the run token before anything is forwarded', async () => { + const s = await standIn(); + await stopping(s.broker, async () => { + const body = { email_address: [newTestEmail(s.run)], skip_password_requirement: true }; + assert.equal((await s.ask('POST', '/users', body, 'not-the-token')).status, 401); + assert.equal((await s.ask('POST', '/users', body, SECRET_KEY)).status, 401); + assert.equal((await fetch(`${s.broker.url}/users`, { method: 'POST', body: JSON.stringify(body) })).status, 401); + assert.deepEqual(s.forwarded, []); + }); + }); + + it('refuses a fourth kind of call: nothing but create a user, mint a ticket, and look a phone up reaches Clerk', async () => { + const s = await standIn(); + await stopping(s.broker, async () => { + const others: readonly (readonly [string, string])[] = [ + ['GET', '/users/count'], + ['GET', '/users/user_created_1'], + ['DELETE', '/users/user_created_1'], + ['PATCH', '/users/user_created_1'], + ['POST', '/users/user_created_1/ban'], + ['POST', '/organizations'], + ['GET', '/sign_in_tokens'], + ['POST', '/sign_in_tokens/sit_1/revoke'], + ['GET', '/instance'], + ['POST', '/users/'], + ]; + for (const [method, path] of others) { + const answer = await s.ask(method, path, method === 'GET' || method === 'DELETE' ? undefined : {}); + assert.equal(answer.status, 404, `${method} ${path}`); + assert.equal(answer.json.errors?.[0]?.code, 'verify_stand_in_unknown_call'); + } + const outsideTheApi = await fetch(`${s.broker.url.replace(/\/v1$/, '')}/users`, { method: 'POST', headers: { Authorization: `Bearer ${readFileSync(s.broker.tokenFile, 'utf8')}` }, body: '{}' }); + assert.equal(outsideTheApi.status, 404); + assert.deepEqual(s.forwarded, []); + }); + }); + + it('creates a user only with one test address of this run, test phones, and the fields the tests send', async () => { + const s = await standIn(); + await stopping(s.broker, async () => { + const mine = newTestEmail(s.run); + const refusedBodies: readonly (readonly [string, Record, RegExp])[] = [ + ['a real address', { email_address: ['someone@example.com'] }, /is not a \+clerk_test email/], + ['an address of another run', { email_address: [newTestEmail('r20200101-000000-aaaa' as RunId)] }, /is not an address of run/], + ['two addresses', { email_address: [mine, newTestEmail(s.run)] }, /exactly one email address/], + ['no address', { phone_number: [TEST_PHONES[0]] }, /exactly one email address/], + ['a real phone', { email_address: [mine], phone_number: ['+14155552671'] }, /is not a 555-0100\.\.0199 test phone/], + ['a phone that is not a list', { email_address: [mine], phone_number: TEST_PHONES[0] }, /not a list of phone numbers/], + ['a phone list that is null', { email_address: [mine], phone_number: null }, /not a list of phone numbers/], + ['metadata', { email_address: [mine], public_metadata: { role: 'admin' } }, /it carries public_metadata/], + ['an external id', { email_address: [mine], external_id: 'x', skip_password_requirement: true }, /it carries external_id/], + ]; + for (const [what, body, why] of refusedBodies) { + const answer = await s.ask('POST', '/users', body); + assert.equal(answer.status, 403, what); + assert.match(whyRefused(answer), why, what); + } + assert.equal((await s.ask('POST', '/users', [mine])).status, 403, 'a body that is not an object'); + assert.deepEqual(s.forwarded, [], 'no refused request reached Clerk'); + assert.equal((await s.ask('POST', '/users', { email_address: [mine], phone_number: [TEST_PHONES[3]], password: 'a-password-Aa1!', bypass_client_trust: true })).status, 200); + assert.equal(s.forwarded.length, 1); + }); + }); + + it('mints a ticket only for a user this run created, and only a short one', async () => { + const s = await standIn(); + await stopping(s.broker, async () => { + const foreign = await s.ask('POST', '/sign_in_tokens', { user_id: 'user_of_someone_else', expires_in_seconds: 120 }); + assert.equal(foreign.status, 403); + assert.match(whyRefused(foreign), /user user_of_someone_else was not created by this run/); + assert.deepEqual(s.forwarded, []); + + const user = await s.users.seed(); + const refusedBodies: readonly (readonly [string, Record, RegExp])[] = [ + ['a long life', { user_id: user.id, expires_in_seconds: 121 }, /120 seconds at most/], + ['a month, which is Clerk\'s default when no life is named', { user_id: user.id }, /120 seconds at most/], + ['a life that is not a number', { user_id: user.id, expires_in_seconds: '60' }, /120 seconds at most/], + ['no user', { expires_in_seconds: 60 }, /was not created by this run/], + ['another field', { user_id: user.id, expires_in_seconds: 60, actor: { sub: 'user_x' } }, /it carries actor/], + ]; + for (const [what, body, why] of refusedBodies) { + const answer = await s.ask('POST', '/sign_in_tokens', body); + assert.equal(answer.status, 403, what); + assert.match(whyRefused(answer), why, what); + } + assert.equal(s.forwarded.length, 1, 'only the create reached Clerk'); + assert.equal((await s.ask('POST', '/sign_in_tokens', { user_id: user.id, expires_in_seconds: 60 })).status, 200); + }); + }); + + it('does not sign in a user that an earlier run created on the same instance', async () => { + const earlier = await standIn(); + const user = await stopping(earlier.broker, () => earlier.users.seed()); + const later = await standIn(); + await stopping(later.broker, async () => { + await assert.rejects(later.users.signInTicket(user), /answered 403: .*was not created by this run; sign in only users from host\.seedUser/); + assert.deepEqual(later.forwarded, []); + }); + }); + + it('looks users up by one test phone only, and tells the test nothing but their ids', async () => { + const s = await standIn((request) => (request.method === 'GET' ? { status: 200, body: [{ id: 'user_7', email_addresses: [{ email_address: 'private@example.com' }], phone_numbers: [{ phone_number: '+12015550107' }] }] } : emptyClerk(request))); + await stopping(s.broker, async () => { + const phone = encodeURIComponent(TEST_PHONES[7]!); + const refusedQueries: readonly (readonly [string, string, RegExp])[] = [ + ['every user', '/users', /looked up by one test phone/], + ['a page of users', '/users?limit=500', /looked up by one test phone/], + ['a real phone', '/users?phone_number=%2B14155552671', /is not a 555-0100\.\.0199 test phone/], + ['a phone and an address', `/users?phone_number=${phone}&email_address=someone@example.com`, /it carries email_address/], + ['a phone and a search', `/users?phone_number=${phone}&query=a`, /it carries query/], + ['two phones', `/users?phone_number=${phone}&phone_number=${phone}`, /looked up by one test phone/], + ]; + for (const [what, path, why] of refusedQueries) { + const answer = await s.ask('GET', path); + assert.equal(answer.status, 403, what); + assert.match(whyRefused(answer), why, what); + } + assert.equal(s.forwarded.length, 0, 'no refused lookup reached Clerk'); + const answer = await s.ask('GET', `/users?phone_number=${phone}`); + assert.deepEqual([answer.status, answer.json], [200, [{ id: 'user_7' }]]); + assert.equal(s.forwarded[0]!.url, `https://api.clerk.com/v1/users?phone_number=${phone}`); + }); + }); + + it('sends Clerk the request it checked, not the text it was sent: one value for each field, each phone in one spelling, and no query on a request that has a body', async () => { + const s = await standIn(); + await stopping(s.broker, async () => { + const mine = newTestEmail(s.run); + const twice = `{"email_address":["someone@example.com"],"email_address":[${JSON.stringify(mine)}],"phone_number":["+1 (201) 555-0100; DROP"],"skip_password_requirement":true}`; + assert.equal((await s.ask('POST', '/users?limit=500&email_address=someone@example.com', twice)).status, 200); + assert.deepEqual([s.forwarded[0]!.url, s.forwarded[0]!.text], ['https://api.clerk.com/v1/users', JSON.stringify({ email_address: [mine], phone_number: ['+12015550100'], skip_password_requirement: true })]); + assert.equal((await s.ask('POST', '/sign_in_tokens?user_id=user_of_someone_else', { user_id: 'user_created_1', expires_in_seconds: 60 })).status, 200); + assert.deepEqual([s.forwarded[1]!.url, s.forwarded[1]!.text], ['https://api.clerk.com/v1/sign_in_tokens', '{"user_id":"user_created_1","expires_in_seconds":60}']); + assert.equal((await s.ask('GET', `/users?phone_number=${encodeURIComponent('(201) 555-0107; DROP')}`)).status, 200); + assert.deepEqual([s.forwarded[2]!.url, s.forwarded[2]!.text], ['https://api.clerk.com/v1/users?phone_number=%2B12015550107', null]); + }); + }); + + it('listens on 127.0.0.1 only, so no other address of this machine reaches it', async (t) => { + const others = Object.values(networkInterfaces()).flatMap((addresses) => addresses ?? []).filter((address) => address.address !== '127.0.0.1' && !address.address.startsWith('fe80:')).map((address) => address.address); + if (others.length === 0) return t.skip('this machine has no other address to try'); + const s = await standIn(); + await stopping(s.broker, async () => { + const port = Number(new URL(s.broker.url).port); + const accepts = (host: string): Promise => + new Promise((resolve) => { + const socket = net.connect({ host, port, timeout: 2_000 }); + const done = (connected: boolean): void => void (socket.destroy(), resolve(connected)); + socket.on('connect', () => done(true)); + socket.on('timeout', () => done(false)); + socket.on('error', () => done(false)); + }); + assert.equal(await accepts('127.0.0.1'), true); + assert.deepEqual(await Promise.all(others.map(accepts)), others.map(() => false), others.join(', ')); + }); + }); + + it('keeps its token in a file that only the user can read', async () => { + const s = await standIn(); + await stopping(s.broker, async () => { + assert.equal(statSync(s.broker.tokenFile).mode & 0o777, 0o600); + }); + }); + + it('refuses a body larger than any request the tests make, and sends none of it to Clerk', async () => { + const s = await standIn(); + await stopping(s.broker, async () => { + const user = (password: string) => ({ email_address: [newTestEmail(s.run)], password, bypass_client_trust: true }); + const large = await s.ask('POST', '/users', user('p'.repeat(64 * 1024))); + assert.equal(large.status, 403); + assert.match(whyRefused(large), /its body is larger than any request the tests make/); + assert.deepEqual(s.forwarded, []); + assert.equal((await s.ask('POST', '/users', user('p'.repeat(1024)))).status, 200); + }); + }); + + it("hands Clerk's own answer back, so a taken phone moves the test to the next one", async () => { + const held = new Set(TEST_PHONES.slice(0, 99)); + const s = await standIn((request) => { + const [phone] = (request.body?.phone_number as string[] | undefined) ?? []; + if (request.method === 'POST' && phone !== undefined && held.has(phone)) return { status: 422, body: { errors: [{ code: 'form_identifier_exists', message: 'That phone number is taken.' }] } }; + return emptyClerk(request); + }); + await stopping(s.broker, async () => { + const taken = await s.ask('POST', '/users', { email_address: [newTestEmail(s.run)], phone_number: [TEST_PHONES[0]] }); + assert.deepEqual([taken.status, taken.json.errors?.[0]?.code], [422, 'form_identifier_exists']); + assert.equal((await s.users.seed({ phone: true })).phone, TEST_PHONES[99]); + }); + }); + + it('puts the fix in the answer when Clerk refuses the key or one user too many', async () => { + const full = await standIn(() => ({ status: 403, body: { errors: [{ code: 'user_quota_exceeded', message: 'quota' }] } })); + await stopping(full.broker, async () => { + await assert.rejects(full.users.seed(), new RegExp(`answered 403: the instance holds the 100 users .* \\(fix: the next \`\\{cli\\} run\` replaces the instance once it holds ${REPLACE_AT_USERS} users; rerun\\)`)); + }); + const replaced = await standIn(() => ({ status: 401, body: { errors: [{ code: 'authentication_invalid' }] } })); + await stopping(replaced.broker, async () => { + await assert.rejects(replaced.users.seed(), /answered 401: api\.clerk\.com answered 401 to the instance's own secret key.*\(fix: set Path prefixes/); + }); + const other = await standIn(() => ({ status: 403, body: { errors: [{ code: 'something_else', long_message: 'Clerk said no.' }] } })); + await stopping(other.broker, async () => { + await assert.rejects(other.users.seed(), /^Error: Clerk POST \/users answered 403: Clerk said no\.$/); + }); + }); + + it('answers with a reason when Clerk cannot be reached', async () => { + const unreachable: Clerk = () => { + throw new Error('fetch failed'); + }; + const s = await standIn(unreachable); + await stopping(s.broker, async () => { + const answer = await s.ask('POST', '/users', { email_address: [newTestEmail(s.run)] }); + assert.equal(answer.status, 502); + assert.match(whyRefused(answer), /could not forward this request: fetch failed/); + }); + }); +}); diff --git a/integration/expo-native/test/busy-runner.test.ts b/integration/expo-native/test/busy-runner.test.ts new file mode 100644 index 00000000000..0d1f93726fa --- /dev/null +++ b/integration/expo-native/test/busy-runner.test.ts @@ -0,0 +1,85 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { describe, it } from 'node:test'; +import { busyWaits, isRunnerBusy, onceTheRunnerIsFree } from '../specs/support/busy-runner.ts'; + +const BUSY_TAP = + 'ENGINE_FAILURE: perform tap failed: the iOS automation runner is still finishing a command that overran its watchdog (session e2e-ios-0 on iPhone 17 Pro): The iOS runner is still finishing a previous command that exceeded its execution watchdog (usually an accessibility capture on a heavy or animating screen). The app is fine. Wait a few seconds and rerun.'; +const BUSY_TYPE = 'agent-device type failed: Error (COMMAND_FAILED): The iOS runner is still finishing a previous command that exceeded its execution watchdog (usually an accessibility capture on a heavy or animating screen).'; +const BUSY_READ = 'APP_UNREACHABLE: snapshot failed: the iOS automation runner is still finishing a command that overran its watchdog (session e2e-ios-0 on iPhone 17 Pro)'; + +const OTHER_ENGINE_FAILURES = [ + "ENGINE_FAILURE: perform tap failed: the iOS automation runner's main thread overran its watchdog on this command (session e2e-ios-0 on iPhone 17 Pro): main thread execution timed out. The app is fine. Rerun once it has drained.", + 'ENGINE_FAILURE: perform tap failed: the iOS automation runner is wedged: its main thread is stuck in abandoned work. agent-device restarts the runner; rerun.', + 'ENGINE_FAILURE: snapshot failed: the iOS automation runner could not present the accessibility snapshot. Rerun.', + 'agent-device type failed: Error (COMMAND_FAILED): main thread execution timed out', + 'snapshot failed: Daemon request timed out', + 'perform tap failed: Ref @e5 is covered by another visible element and cannot be tapped safely', + 'LOCATOR_NOT_FOUND', +]; + +function step(outcomes: readonly (string | null)[]) { + let ran = 0; + let waited = 0; + return { + ran: () => ran, + waited: () => waited, + run: async () => { + const outcome = outcomes[Math.min(ran, outcomes.length - 1)]!; + ran += 1; + if (outcome !== null) throw new Error(outcome); + return 'done'; + }, + busy: (most: number) => busyWaits(most, async () => void (waited += 1)), + }; +} + +describe('a step the iOS runner refuses because it is still finishing an earlier command', () => { + it('knows the refusal as the engine words it for a tap and a screen read, and as agent-device words it for typing', () => { + for (const message of [BUSY_TAP, BUSY_TYPE, BUSY_READ]) assert.equal(isRunnerBusy(new Error(message)), true, message); + }); + + it('takes no other failure of the engine or of agent-device for it', () => { + for (const message of OTHER_ENGINE_FAILURES) assert.equal(isRunnerBusy(new Error(message)), false, message); + assert.equal(isRunnerBusy(BUSY_TAP), false, 'a thrown string is not an engine failure'); + }); + + it('runs again after a wait and passes when the second try succeeds', async () => { + const tap = step([BUSY_TAP, null]); + assert.equal(await onceTheRunnerIsFree(tap.run, tap.busy(6)), 'done'); + assert.equal(tap.ran(), 2); + assert.equal(tap.waited(), 1); + }); + + it('runs once and never waits when the runner is free', async () => { + const tap = step([null]); + await onceTheRunnerIsFree(tap.run, tap.busy(6)); + assert.deepEqual([tap.ran(), tap.waited()], [1, 0]); + }); + + it('stops at the bound and fails with the first refusal', async () => { + const tap = step([BUSY_TAP, BUSY_READ]); + await assert.rejects(onceTheRunnerIsFree(tap.run, tap.busy(6)), (error: Error) => error.message === BUSY_TAP); + assert.deepEqual([tap.ran(), tap.waited()], [7, 6]); + }); + + it('gives the steps that share one bound that many waits in all, so a runner that stays busy fails each later step at its first refusal', async () => { + const tap = step([BUSY_TAP]); + const read = step([BUSY_READ]); + const oneCall = tap.busy(6); + await assert.rejects(onceTheRunnerIsFree(tap.run, oneCall), (error: Error) => error.message === BUSY_TAP); + await assert.rejects(onceTheRunnerIsFree(read.run, oneCall), (error: Error) => error.message === BUSY_READ); + assert.deepEqual([tap.ran(), read.ran(), tap.waited()], [7, 1, 6]); + }); + + it('does not run again after any other failure, on the first try or on a later one', async () => { + for (const message of OTHER_ENGINE_FAILURES) { + const first = step([message, null]); + await assert.rejects(onceTheRunnerIsFree(first.run, first.busy(6)), (error: Error) => error.message === message); + assert.deepEqual([first.ran(), first.waited()], [1, 0], message); + } + const later = step([BUSY_TAP, 'LOCATOR_NOT_FOUND', null]); + await assert.rejects(onceTheRunnerIsFree(later.run, later.busy(6)), /LOCATOR_NOT_FOUND/); + assert.deepEqual([later.ran(), later.waited()], [2, 1]); + }); +}); diff --git a/integration/expo-native/test/claims.test.ts b/integration/expo-native/test/claims.test.ts new file mode 100644 index 00000000000..e73f48006b4 --- /dev/null +++ b/integration/expo-native/test/claims.test.ts @@ -0,0 +1,48 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { execFile } from 'node:child_process'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { freeSlot, isOrphaned, readClaim, takeSlot } from '../src/core/claims.ts'; +import { currentProcess, isRunning } from '../src/core/exec.ts'; + +const taker = join(import.meta.dirname, '..', 'testing', 'claim-taker.ts'); + +function take(dir: string, worktree: string, startAt: number, from: number): Promise { + return new Promise((resolve, reject) => { + execFile(process.execPath, [taker, dir, worktree, String(startAt), String(from)], (error, stdout) => (error === null ? resolve(stdout) : reject(error))); + }); +} + +describe('device claims', () => { + it('lets only one of several processes take an orphaned slot', async () => { + for (let round = 0; round < 4; round += 1) { + const dir = mkdtempSync(join(tmpdir(), 'verify-claims-')); + const gone = mkdtempSync(join(tmpdir(), 'verify-gone-')); + const orphan = takeSlot(dir, 'ios', 1, 0, gone)!; + rmSync(gone, { recursive: true }); + assert.equal(isOrphaned({ ...orphan, owner: { pid: 1, startedAt: 0 } }), true); + const startAt = Date.now() + 1500; + const results = await Promise.all(Array.from({ length: 6 }, (_, i) => take(dir, join(dir, `worktree-${i}`), startAt, orphan.gen))); + assert.equal(results.filter((r) => r === 'won').length, 1, `round ${round}: ${results.join(' ')}`); + } + }); + + it('never lets a stale holder overwrite or free a slot another worktree now holds', () => { + const dir = mkdtempSync(join(tmpdir(), 'verify-claims-')); + const first = takeSlot(dir, 'ios', 1, 0, '/worktrees/a')!; + assert.equal(freeSlot(dir, first), true); + const second = takeSlot(dir, 'ios', 1, readClaim(dir, 'ios', 1).gen, '/worktrees/b')!; + assert.equal(takeSlot(dir, 'ios', 1, first.gen, '/worktrees/a'), null, 'a retake from a stale generation fails'); + assert.equal(freeSlot(dir, first), false, 'a stale free fails'); + assert.equal(readClaim(dir, 'ios', 1).claim?.nonce, second.nonce); + }); + + it('treats a reused pid as a different process', () => { + const me = currentProcess(); + assert.equal(isRunning(me), true); + assert.equal(isRunning({ pid: me.pid, startedAt: me.startedAt - 3_600_000 }), false); + }); +}); diff --git a/integration/expo-native/test/clerk.test.ts b/integration/expo-native/test/clerk.test.ts new file mode 100644 index 00000000000..d77b1ab6cf1 --- /dev/null +++ b/integration/expo-native/test/clerk.test.ts @@ -0,0 +1,58 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { describe, it } from 'node:test'; +import { runEmailPrefix } from '../specs/support/clerk.ts'; +import { Secret } from '../specs/support/secret.ts'; +import { createClerkBackends } from '../src/core/clerk.ts'; +import { VerifyFailure, type PublishableKey, type RunId } from '../src/core/types.ts'; + +const keys = { pk: 'pk_test_x' as PublishableKey, sk: new Secret('clerk-secret-key', 'sk_test_x') }; + +function fakeBapi(answer: { status: number; body: unknown }) { + const calls: { readonly url: string; readonly signal: AbortSignal | null | undefined }[] = []; + const fetchImpl = (async (url: string | URL, init?: RequestInit) => { + calls.push({ url: `${init?.method ?? 'GET'} ${String(url)}`, signal: init?.signal }); + return new Response(JSON.stringify(answer.body), { status: answer.status }); + }) as typeof fetch; + return { calls, backend: createClerkBackends(fetchImpl)(() => keys) }; +} + +describe('what the CLI asks a development instance about its users', () => { + const run = 'r20261003-000000-abcd' as RunId; + const prefix = runEmailPrefix(run); + + it('counts them with the instance\'s own key', async () => { + const { calls, backend } = fakeBapi({ status: 200, body: { object: 'total_count', total_count: 61 } }); + assert.equal(await backend.userCount(), 61); + assert.deepEqual(calls.map((call) => call.url), ['GET https://api.clerk.com/v1/users/count']); + }); + + it('gives each call a time limit, so a stalled API cannot hold the instances lock for good', async () => { + const { calls, backend } = fakeBapi({ status: 200, body: { object: 'total_count', total_count: 0 } }); + await backend.userCount(); + assert.ok(calls[0]!.signal instanceof AbortSignal); + }); + + it('lists the users of one run by the start of their addresses, in one request, whether a test seeded them or signed them up in the app', async () => { + const { calls, backend } = fakeBapi({ + status: 200, + body: [ + { id: 'user_1', email_addresses: [{ email_address: `${prefix}0a1b2c3d+clerk_test@example.com` }] }, + { id: 'user_2', email_addresses: [{ email_address: 'someone@example.com' }, { email_address: `${prefix}ffffffff+clerk_test@example.com` }] }, + { id: 'user_3', email_addresses: [{ email_address: `not_${prefix}00000000+clerk_test@example.com` }] }, + { id: 'user_4', email_addresses: [] }, + { email_addresses: [{ email_address: `${prefix}11111111+clerk_test@example.com` }] }, + ], + }); + assert.deepEqual(await backend.usersOfRun(run), [ + { userId: 'user_1', email: `${prefix}0a1b2c3d+clerk_test@example.com` }, + { userId: 'user_2', email: `${prefix}ffffffff+clerk_test@example.com` }, + ]); + assert.deepEqual(calls.map((call) => call.url), [`GET https://api.clerk.com/v1/users?limit=500&email_address_query=${prefix}`]); + }); + + it('says what to change when Clerk refuses the instance\'s own key, and names the request when it fails another way', async () => { + await assert.rejects(fakeBapi({ status: 401, body: {} }).backend.usersOfRun(run), (error: VerifyFailure) => error instanceof VerifyFailure && error.code === 'NOT_READY' && /Path prefixes/.test(error.fix)); + await assert.rejects(fakeBapi({ status: 500, body: { errors: [{ code: 'internal' }] } }).backend.usersOfRun(run), /^Error: Clerk GET \/users answered 500 \(internal\)$/); + }); +}); diff --git a/integration/expo-native/test/cli.test.ts b/integration/expo-native/test/cli.test.ts new file mode 100644 index 00000000000..b39bd2ecab7 --- /dev/null +++ b/integration/expo-native/test/cli.test.ts @@ -0,0 +1,248 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { describe, it } from 'node:test'; +import { createOutput, exitCodeFor, parseArgv } from '../src/core/cli.ts'; +import { Secret } from '../specs/support/secret.ts'; +import { supportsNode } from '../src/core/launch.mjs'; +import { VerifyFailure, type Command, type DoctorReport, type DownResult, type EvidencePath, type EvidenceRecord, type RunResult, type SpecResult } from '../src/core/types.ts'; + +function usageError(argv: readonly string[]): VerifyFailure { + try { + parseArgv(argv); + } catch (error) { + assert.ok(error instanceof VerifyFailure, `${argv.join(' ')} threw a non-VerifyFailure`); + return error; + } + assert.fail(`${argv.join(' ')} parsed`); +} + +describe('parseArgv', () => { + it('parses every verb', () => { + assert.deepEqual(parseArgv(['doctor']).command, { verb: 'doctor', live: false }); + assert.deepEqual(parseArgv(['doctor', '--platform', 'ios', '--backend', 'local']).command, { verb: 'doctor', platform: 'ios', backend: 'local', live: false }); + assert.deepEqual(parseArgv(['up', '--backend', 'auto']).command, { verb: 'up', waitSeconds: 0 }); + assert.throws(() => parseArgv(['up', '--backend', 'elsewhere']), /auto or local/); + assert.deepEqual(parseArgv(['up', '--wait', '300']).command, { verb: 'up', waitSeconds: 300 }); + assert.deepEqual(parseArgv(['up']).command, { verb: 'up', waitSeconds: 0 }); + assert.deepEqual(parseArgv(['run', 'auth-start', 'sign-up/request-code']).command, { + verb: 'run', + selection: { selectors: ['auth-start', 'sign-up/request-code'] }, + video: true, + retries: 0, + githubReport: false, + waitSeconds: 0, + }); + assert.deepEqual(parseArgv(['run', '--all', '--no-video', '--grep', 'profile']).command, { + verb: 'run', + selection: { all: true }, + grep: 'profile', + video: false, + retries: 0, + githubReport: false, + waitSeconds: 0, + }); + assert.equal((parseArgv(['run', 'auth-start', '--wait', '300']).command as { waitSeconds: number }).waitSeconds, 300); + assert.equal((parseArgv(['run', '--all', '--retries', '1']).command as Extract).retries, 1); + assert.equal((parseArgv(['run', '--all', '--github-report']).command as Extract).githubReport, true); + for (const bad of ['-1', 'once']) assert.throws(() => parseArgv(['run', '--all', '--retries', bad]), { code: 'USAGE' }, bad); + assert.deepEqual(parseArgv(['screen', '--png']).command, { verb: 'screen', png: true }); + assert.deepEqual(parseArgv(['attach', 'r20261002-141210-7c1e', '--pr', '412', '--screenshot', 'a', '--screenshot=b']).command, { + verb: 'attach', + run: 'r20261002-141210-7c1e', + pr: 412, + screenshots: ['a', 'b'], + }); + assert.deepEqual(parseArgv(['attach', 'r20261002-141210-7c1e', '--pr=7']).command, { verb: 'attach', run: 'r20261002-141210-7c1e', pr: 7, screenshots: 'all' }); + assert.deepEqual(parseArgv(['down', '--stale', '--dry-run']).command, { verb: 'down', stale: true, dryRun: true }); + }); + + it('reads --json on every verb', () => { + for (const argv of [['doctor'], ['up'], ['run', 'x'], ['screen'], ['attach', 'r20261002-141210-7c1e', '--pr', '1'], ['down']]) { + assert.equal(parseArgv([...argv, '--json']).json, true, argv[0]); + assert.equal(parseArgv(argv).json, false, argv[0]); + } + }); + + it('refuses unknown verbs, flags, and malformed values with USAGE', () => { + for (const argv of [ + [], + ['frobnicate'], + ['Doctor'], + ['doctor', '--wait', '3'], + ['up', '--png'], + ['run'], + ['run', 'x', '--all'], + ['run', 'x', '--video'], + ['screen', 'extra'], + ['attach', 'r20261002-141210-7c1e'], + ['attach', 'not-a-run', '--pr', '1'], + ['attach', 'r20261002-141210-7c1e', '--pr', 'abc'], + ['down', '--platform', 'windows'], + ['up', '--backend', 'cloud'], + ['up', '--wait'], + ]) { + assert.equal(usageError(argv).code, 'USAGE', argv.join(' ')); + } + }); +}); + +describe('Output', () => { + it('masks every value a secret was used with, in human and JSON output', () => { + const sk = new Secret('clerk-secret-key', 'sk_test_unitTestValue123456'); + sk.use('bapi-authorization', () => undefined); + let written = ''; + const sink = { write: (text: string) => (written += text) }; + const human = createOutput(false, '/tmp', 'bin/control-x', sink, sink); + human.progress('calling with sk_test_unitTestValue123456 now'); + human.failure(new VerifyFailure('NOT_READY', 'header was Bearer sk_test_unitTestValue123456', 'retry')); + const json = createOutput(true, '/tmp', 'bin/control-x', sink, sink); + json.failure(new VerifyFailure('NOT_READY', 'sk_test_unitTestValue123456', 'retry')); + assert.ok(!written.includes('sk_test_unitTestValue123456'), written); + assert.equal(written.match(//g)?.length, 3); + assert.equal(String(sk), ''); + assert.equal(JSON.stringify({ sk }), '{"sk":""}'); + }); + + it('prints one envelope under --json and keeps progress off stdout', () => { + let out = ''; + let err = ''; + const output = createOutput(true, '/tmp', 'bin/control-x', { write: (t: string) => (out += t) }, { write: (t: string) => (err += t) }); + const report: DoctorReport = { verb: 'doctor', ok: false, backend: { ios: 'local' }, checks: [{ id: 'build', ok: false, detail: 'none', fix: '{cli} up' }] }; + output.progress('building'); + output.result(report); + assert.equal(err, ''); + const parsed = JSON.parse(out) as { ok: boolean; checks: unknown[] }; + assert.equal(parsed.ok, false); + assert.equal(parsed.checks.length, 1); + assert.equal(exitCodeFor(report), 3); + }); +}); + +describe('run output', () => { + const result = (status: SpecResult['status'], error: string): RunResult => ({ + verb: 'run', + dir: '/tmp/runs/r20261002-141210-7c1e' as EvidencePath, + next: 'next', + record: { + results: [ + { spec: { kind: 'golden', path: 'specs/golden/auth-start/opens.e2e.ts', feature: null }, title: 'opens', platform: 'ios', status: 'passed', seconds: 5, attempts: 1, error: null, skipReason: null, skippedBy: null, tags: [], failurePage: null, failureScreen: null, failureScreenshot: null }, + { spec: { kind: 'golden', path: 'specs/golden/sign-up/complete.e2e.ts', feature: null }, title: 'completes', platform: 'ios', status, seconds: 41.5, attempts: 2, error, skipReason: null, skippedBy: null, tags: [], failurePage: '/tmp/failures/complete.md' as EvidencePath, failureScreen: null, failureScreenshot: null }, + ], + videos: [], + screenshots: [], + appLog: null, + tainted: [], + } as unknown as EvidenceRecord, + }); + const printed = (value: RunResult): string => { + let out = ''; + createOutput(false, '/tmp', 'bin/control-x', { write: (t: string) => (out += t) }, { write: () => {} }).result(value); + return out; + }; + + it('prints a test that passed on a retry as flaky with the failed attempt, counts it, and does not fail the run', () => { + const flaky = result('flaky', 'tap failed'); + const out = printed(flaky); + assert.match(out, /^ flaky sign-up\/complete\.e2e\.ts +completes 41\.5s$/m); + assert.match(out, /^ +passed on attempt 2; the attempt before it failed: tap failed$/m); + assert.match(out, /^ +failure page .*complete\.md$/m); + assert.match(out, /^flaky 1 test passed only on a retry, and the run does not fail for it$/m); + assert.doesNotMatch(out, /^ pass sign-up/m); + assert.equal(exitCodeFor(flaky), 0); + }); + + it('fails the run for a test that failed on every attempt', () => { + const failed = result('failed', 'tap failed again'); + const out = printed(failed); + assert.match(out, /^ FAIL sign-up\/complete\.e2e\.ts +completes 41\.5s$/m); + assert.match(out, /^ +tap failed again$/m); + assert.doesNotMatch(out, /^flaky/m); + assert.equal(exitCodeFor(failed), 1); + }); +}); + +describe('doctor output', () => { + const report: DoctorReport = { + verb: 'doctor', + ok: true, + backend: { ios: 'local' }, + checks: [ + { id: 'node', ok: true, detail: '24.15.0' }, + { id: 'gh-attach', ok: true, state: 'warning', detail: 'gh pr comment has no --attach', fix: 'install one that has' }, + { id: 'live-session', ok: true, state: 'not-run', detail: 'not run: needs --live' }, + ], + }; + + it('labels a warning and a check that was not run, and neither fails doctor', () => { + let out = ''; + createOutput(false, '/tmp', 'bin/control-x', { write: (t: string) => (out += t) }, { write: () => true }).result(report); + assert.deepEqual(out.trimEnd().split('\n'), ['ok node 24.15.0', 'warn gh-attach gh pr comment has no --attach', ' fix: install one that has', 'skip live-session not run: needs --live']); + assert.equal(exitCodeFor(report), 0); + }); + + it('carries the state of each check under --json', () => { + let out = ''; + createOutput(true, '/tmp', 'bin/control-x', { write: (t: string) => (out += t) }, { write: () => true }).result(report); + assert.deepEqual((JSON.parse(out) as DoctorReport).checks.map((c) => c.state ?? null), [null, 'warning', 'not-run']); + }); +}); + +describe('doctor node check', () => { + it('accepts Node 24 from 24.8, which is the oldest 24 that the pinned e2e runs on', () => { + assert.deepEqual( + ['24.8.0', '24.15.0', '24.21.0'].map(supportsNode), + [true, true, true], + ); + assert.deepEqual( + ['24.7.9', '24.0.0', '22.22.3', '25.0.0', '26.1.0'].map(supportsNode), + [false, false, false, false, false], + ); + }); +}); + +describe('down output', () => { + const text = (result: DownResult) => { + let out = ''; + createOutput(false, '/tmp', 'bin/control-x', { write: (t: string) => (out += t) }, { write: () => true }).result(result); + return out; + }; + const render = (stoppedProcesses: readonly string[]) => text({ verb: 'down', dryRun: false, released: [], deletedApplications: [{ name: 'verify-throwaway-one' }], stoppedProcesses, keptRuns: [] }); + + it('says a ledgered daemon had already exited instead of claiming none ran', () => { + const out = render(['agent-device 4242 had already exited']); + assert.match(out, /stopped agent-device 4242 had already exited/); + assert.doesNotMatch(out, /no agent-device daemon running/); + assert.match(out, /deleted 1 application \(verify-throwaway-one, with every test user in it\)/); + }); + + it('names each application a dry run would delete', () => { + const out = text({ verb: 'down', dryRun: true, wouldRelease: [], wouldDelete: [{ kind: 'application', name: 'verify-throwaway-one' }], wouldStop: [], keptRuns: [] }); + assert.match(out, /would delete 1 application\n application verify-throwaway-one \(with every test user in it\)\n/); + }); + + it('says no daemon ran when the ledger had none', () => { + assert.match(render([]), /stopped nothing; no agent-device daemon running/); + }); +}); + +describe('the CLI name', () => { + it('prints fixes, usage, and next hints with the host command, never a built-in name', () => { + let out = ''; + const sink = { write: (t: string) => (out += t) }; + const output = createOutput(false, '/tmp', 'tools/bin/control-acme', sink, sink); + output.failure(usageError(['run'])); + output.failure(new VerifyFailure('NOT_READY', 'no device is leased', '{cli} up')); + output.progress('wait another {cli} run in this worktree is driving the device'); + assert.match(out, /fix: tools\/bin\/control-acme up/); + assert.match(out, /tools\/bin\/control-acme run (json += t) }, sink).failure(new VerifyFailure('NOT_READY', 'x', '{cli} doctor')); + assert.equal((JSON.parse(json) as { error: { fix: string } }).error.fix, 'control-acme doctor'); + }); +}); diff --git a/integration/expo-native/test/device-command.test.ts b/integration/expo-native/test/device-command.test.ts new file mode 100644 index 00000000000..4a13b6ed239 --- /dev/null +++ b/integration/expo-native/test/device-command.test.ts @@ -0,0 +1,52 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { mkdirSync, mkdtempSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { deviceCommand, deviceToolCommand, type CommandRunner } from '../specs/support/device.ts'; +import type { Device } from '../specs/support/inputs.ts'; + +describe('the device tool', () => { + it('is adb on the device for shell and for reverse to a TCP port, and nothing that reaches the files or sockets of the machine adb runs on', () => { + assert.deepEqual(deviceToolCommand('android', 'emulator-5560', ['shell', 'am force-stop com.x']), { command: 'adb', args: ['-s', 'emulator-5560', 'shell', 'am force-stop com.x'] }); + assert.deepEqual(deviceToolCommand('android', 'emulator-5560', ['reverse', 'tcp:8081', 'tcp:8081'], '/sdk/adb')?.command, '/sdk/adb'); + for (const args of [['reverse', '--list'], ['reverse', '--remove', 'tcp:8081'], ['reverse', '--no-rebind', 'tcp:8081', 'tcp:8081']]) assert.notEqual(deviceToolCommand('android', 'emulator-5560', args), null, args.join(' ')); + const refused = [['pull', '/sdcard/x', '/etc/x'], ['push', '/etc/passwd', '/sdcard/x'], ['emu', 'kill'], ['-s', 'other', 'shell'], [], ['reverse', 'tcp:9000', 'localfilesystem:/var/run/docker.sock'], ['reverse', 'tcp:9000', 'localabstract:x'], ['reverse', '--unknown']]; + for (const args of refused) assert.equal(deviceToolCommand('android', 'emulator-5560', args), null, args.join(' ')); + assert.equal(deviceToolCommand('ios', 'UDID', ['shell', 'id']), null); + }); +}); + +describe('deviceCommand', () => { + it('runs adb from the SDK on this machine for a local device', async () => { + const sdk = mkdtempSync(join(tmpdir(), 'verify-sdk-')); + const ran: unknown[] = []; + const device: Device = { kind: 'local', id: 'emulator-5560' }; + const runner: CommandRunner = async (command, args) => (ran.push([command, args]), { code: 0, stdout: 'ok', stderr: '' }); + const home = mkdtempSync(join(tmpdir(), 'verify-home-')); + assert.deepEqual(await deviceCommand(device, 'android', ['shell', 'ls'], { runner, env: { ANDROID_HOME: sdk }, home }), { code: 0, stdout: 'ok', stderr: '' }); + assert.deepEqual(ran, [['adb', ['-s', 'emulator-5560', 'shell', 'ls']]], 'an SDK with no adb falls back to PATH'); + mkdirSync(join(sdk, 'platform-tools')); + writeFileSync(join(sdk, 'platform-tools', 'adb'), ''); + await deviceCommand(device, 'android', ['shell', 'id'], { runner, env: { ANDROID_HOME: sdk }, home }); + assert.deepEqual(ran[1], [join(sdk, 'platform-tools', 'adb'), ['-s', 'emulator-5560', 'shell', 'id']]); + }); + + it('finds adb in the SDK that Android Studio installs when no variable names one', async () => { + const home = mkdtempSync(join(tmpdir(), 'verify-home-')); + const studioSdk = process.platform === 'darwin' ? join(home, 'Library', 'Android', 'sdk') : join(home, 'Android', 'Sdk'); + mkdirSync(join(studioSdk, 'platform-tools'), { recursive: true }); + writeFileSync(join(studioSdk, 'platform-tools', 'adb'), ''); + const ran: unknown[] = []; + const device: Device = { kind: 'local', id: 'emulator-5560' }; + const runner: CommandRunner = async (command, args) => (ran.push([command, args]), { code: 0, stdout: '', stderr: '' }); + await deviceCommand(device, 'android', ['shell', 'id'], { runner, env: { ANDROID_HOME: '' }, home }); + assert.deepEqual(ran, [[join(studioSdk, 'platform-tools', 'adb'), ['-s', 'emulator-5560', 'shell', 'id']]]); + }); + + it('refuses arguments that are not a device command, before anything runs', async () => { + const device: Device = { kind: 'local', id: 'emulator-5560' }; + await assert.rejects(deviceCommand(device, 'android', ['pull', '/sdcard/x', 'x'], { runner: async () => assert.fail('ran') }), /no android device command starts with pull/); + }); +}); diff --git a/integration/expo-native/test/down.test.ts b/integration/expo-native/test/down.test.ts new file mode 100644 index 00000000000..20643614334 --- /dev/null +++ b/integration/expo-native/test/down.test.ts @@ -0,0 +1,161 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { spawn, type ChildProcess } from 'node:child_process'; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { heldInstances } from '../testing/fake-instances.ts'; +import { describe, it } from 'node:test'; +import { openProcesses } from '../src/core/ledgers.ts'; +import { down, type Deps } from '../src/core/verbs.ts'; +import { newEntryId, openWorkspace } from '../src/core/workspace.ts'; +import type { DeviceBackend, HostAdapter, LocalLease, Platform } from '../src/core/types.ts'; + +function setup() { + const packageDir = mkdtempSync(join(tmpdir(), 'verify-down-')); + const workspace = openWorkspace({ packageDir, worktree: packageDir, home: join(packageDir, 'home') }); + const released: string[] = []; + const finished: string[] = []; + const backend = { + kind: 'local', + platform: 'ios', + availability: () => ({ usable: true, why: 'test' }), + release: async (lease: LocalLease) => void released.push(lease.deviceId), + reapable: async () => [], + describe: (lease: LocalLease) => lease.deviceName, + } as unknown as DeviceBackend; + const host = { repo: 'clerk-ios', platforms: ['ios'], backends: [backend] } as unknown as HostAdapter; + const instances = heldInstances({ finish: async (ledger) => (finished.push(ledger.worktree), []) }); + const deps: Deps = { host, workspace, runner: async () => assert.fail('down runs no commands'), env: {}, progress: () => undefined, instances }; + + const lease: LocalLease = { backend: 'local', platform: 'ios', slot: 2, deviceName: 'verify-ios-2', deviceId: 'UDID-2', claimNonce: 'claim-2', acquiredAt: '2026-10-03T00:00:00Z', installedBuild: null }; + workspace.writeLease(lease); + workspace.append({ id: newEntryId(), kind: 'process', what: 'watch', pid: 2147483646, startedAt: '2026-10-03T00:00:00Z' }); + const { run } = workspace.newRun(); + return { deps, workspace, released, finished, run }; +} + +describe('down', () => { + it('--dry-run reports the plan and changes nothing', async () => { + const { deps, workspace, released, finished, run } = setup(); + const ledgerBefore = readFileSync(workspace.ledgerFile, 'utf8'); + const result = await down(deps, { verb: 'down', stale: false, dryRun: true }); + assert.ok(result.dryRun); + assert.equal('released' in result, false, 'a dry run reports nothing in the past tense'); + assert.deepEqual(result.wouldRelease.map((l) => l.device), ['verify-ios-2']); + assert.deepEqual(result.keptRuns, [run]); + assert.deepEqual(released, []); + assert.deepEqual(finished, []); + assert.ok(existsSync(workspace.leaseFile('ios')), 'lease file kept'); + assert.equal(readFileSync(workspace.ledgerFile, 'utf8'), ledgerBefore, 'ledger untouched'); + }); + + it('releases the lease, finishes the instances of this worktree, tombstones, and keeps runs', async () => { + const { deps, workspace, released, finished, run } = setup(); + const result = await down(deps, { verb: 'down', stale: false, dryRun: false }); + assert.ok(!result.dryRun); + assert.deepEqual(released, ['UDID-2']); + assert.deepEqual(finished, [workspace.worktree]); + assert.deepEqual(result.stoppedProcesses, ['watch 2147483646 had already exited']); + assert.equal(workspace.readLease('ios'), null); + assert.deepEqual(workspace.unclosedEntries(), []); + assert.ok(existsSync(workspace.runDir(run)), 'evidence survives'); + const again = await down(deps, { verb: 'down', stale: false, dryRun: false }); + assert.ok(!again.dryRun); + assert.deepEqual(again.released, []); + }); +}); + +describe('down --stale', () => { + it('asks a backend this machine can no longer use, because a claim made before that still has to be finished', async () => { + const { deps } = setup(); + const orphan: LocalLease = { backend: 'local', platform: 'ios', slot: 1, deviceName: 'verify-ios-1', deviceId: 'UDID-1', claimNonce: 'claim-1', acquiredAt: '', installedBuild: null }; + const unusable = { ...deps.host.backends[0]!, availability: () => ({ usable: false, why: 'the SDK was removed' }), reapable: async () => [orphan] } as DeviceBackend; + const result = await down({ ...deps, host: { ...deps.host, backends: [unusable] } }, { verb: 'down', stale: true, dryRun: true }); + assert.ok(result.dryRun); + assert.deepEqual(result.wouldRelease.map((l) => l.device), ['verify-ios-2', 'verify-ios-1']); + }); +}); + +describe('down --platform on a host with two platforms', () => { + function twoPlatforms(leased: readonly Platform[]) { + const packageDir = mkdtempSync(join(tmpdir(), 'verify-down-two-')); + const workspace = openWorkspace({ packageDir, worktree: packageDir, home: join(packageDir, 'home') }); + const backends = (['ios', 'android'] as const).map( + (platform) => ({ kind: 'local', platform, availability: () => ({ usable: true, why: 'test' }), release: async () => undefined, reapable: async () => [], describe: (lease: LocalLease) => lease.deviceName }) as unknown as DeviceBackend, + ); + const host = { repo: 'clerk-expo', platforms: ['ios', 'android'], backends } as unknown as HostAdapter; + const keptApplications: boolean[] = []; + const instances = heldInstances({ finish: async (_ledger, options) => (keptApplications.push(options.keepApplications), []) }); + const deps: Deps = { host, workspace, runner: async () => assert.fail('down runs no commands'), env: {}, progress: () => undefined, instances }; + for (const platform of leased) { + workspace.writeLease({ backend: 'local', platform, slot: 1, deviceName: `verify-${platform}-1`, deviceId: `${platform}-device`, claimNonce: `claim-${platform}`, acquiredAt: '2026-10-03T00:00:00Z', installedBuild: null }); + } + workspace.append({ id: newEntryId(), kind: 'application', name: 'verify-throwaway-two', workspace: 'a workspace' }); + const children: ChildProcess[] = []; + const start = (): { readonly pid: number; readonly startedAt: string } => { + const child = spawn('sleep', ['30'], { stdio: 'ignore' }); + children.push(child); + return { pid: child.pid!, startedAt: new Date().toISOString() }; + }; + const ledgered = (what: 'metro' | 'watch', platform?: Platform): string => { + const { pid, startedAt } = start(); + workspace.append({ id: newEntryId(), kind: 'process', what, pid, startedAt, ...(platform === undefined ? {} : { platform }) }); + return `${what} ${pid}`; + }; + const watch = ledgered('watch'); + const iosMetro = ledgered('metro', 'ios'); + const androidMetro = ledgered('metro', 'android'); + const daemon = start(); + mkdirSync(workspace.agentDeviceDir, { recursive: true }); + writeFileSync(join(workspace.agentDeviceDir, 'daemon.json'), JSON.stringify({ pid: daemon.pid, processStartTime: daemon.startedAt })); + const exited = async (line: string, withinMs: number): Promise => { + const child = children.find((c) => c.pid === Number(line.split(' ')[1]))!; + for (let waited = 0; waited < withinMs && child.exitCode === null && child.signalCode === null; waited += 20) await new Promise((resolve) => setTimeout(resolve, 20)); + return child.exitCode !== null || child.signalCode !== null; + }; + const stopped = (line: string) => exited(line, 2000); + const stillRuns = async (...lines: string[]) => (await Promise.all(lines.map((line) => exited(line, 200)))).every((gone) => !gone); + const stop = (platform: Platform, dryRun: boolean) => down(deps, { verb: 'down', platform, stale: false, dryRun }); + return { workspace, keptApplications, watch, iosMetro, androidMetro, daemon: `agent-device ${daemon.pid}`, stopped, stillRuns, stop, cleanup: () => children.forEach((child) => child.kill()) }; + } + + it('stops only the processes of the platform asked for while the other platform stays leased, and a dry run lists the same', async () => { + const w = twoPlatforms(['ios', 'android']); + try { + const planned = await w.stop('ios', true); + assert.ok(planned.dryRun); + assert.deepEqual(planned.wouldStop, [w.iosMetro]); + assert.deepEqual(planned.wouldDelete, [], 'the android lease still uses the application'); + const result = await w.stop('ios', false); + assert.ok(!result.dryRun); + assert.deepEqual(result.stoppedProcesses, planned.wouldStop); + assert.equal(await w.stopped(w.iosMetro), true); + assert.equal(await w.stillRuns(w.watch, w.androidMetro, w.daemon), true, 'the watch build, the other Metro, and the daemon keep running'); + assert.deepEqual(openProcesses(w.workspace).map((entry) => `${entry.what} ${entry.pid}`), [w.watch, w.androidMetro, w.daemon]); + assert.deepEqual(w.keptApplications, [true]); + assert.notEqual(w.workspace.readLease('android'), null); + } finally { + w.cleanup(); + } + }); + + it('stops what every lease shares once no other platform is leased, and leaves a process of a platform it was not asked about', async () => { + const w = twoPlatforms(['android']); + try { + const planned = await w.stop('android', true); + assert.ok(planned.dryRun); + assert.deepEqual(planned.wouldStop, [w.watch, w.androidMetro, w.daemon]); + assert.deepEqual(planned.wouldDelete, [{ kind: 'application', name: 'verify-throwaway-two' }]); + const result = await w.stop('android', false); + assert.ok(!result.dryRun); + assert.deepEqual(result.stoppedProcesses, planned.wouldStop); + assert.deepEqual([await w.stopped(w.watch), await w.stopped(w.androidMetro), await w.stopped(w.daemon)], [true, true, true]); + assert.equal(await w.stillRuns(w.iosMetro), true); + assert.deepEqual(openProcesses(w.workspace).map((entry) => `${entry.what} ${entry.pid}`), [w.iosMetro]); + assert.deepEqual(w.keptApplications, [false]); + } finally { + w.cleanup(); + } + }); +}); diff --git a/integration/expo-native/test/driver.test.ts b/integration/expo-native/test/driver.test.ts new file mode 100644 index 00000000000..bad85bb64b1 --- /dev/null +++ b/integration/expo-native/test/driver.test.ts @@ -0,0 +1,629 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { randomBytes } from 'node:crypto'; +import { appendFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { DRIVER_LOG_LIMIT_BYTES, LEFT_OUT, clearStuckRegistration, linesLeftOut, readRegistration, scrubDriverLog, watchDriver } from '../src/core/driver.ts'; +import type { EvidencePath } from '../src/core/types.ts'; +import { protect } from '../specs/support/secret.ts'; + +const GONE = 2147483646; +const stateDir = (registration?: unknown): string => { + const dir = mkdtempSync(join(tmpdir(), 'verify-driver-')); + if (registration !== undefined) writeFileSync(join(dir, 'daemon.json'), typeof registration === 'string' ? registration : JSON.stringify(registration)); + return dir; +}; +const nothingRuns = (): boolean => false; +const everythingRuns = (): boolean => true; + +describe('the agent-device registration a run finds', () => { + it('is none when the state directory holds no daemon.json', () => { + assert.deepEqual(clearStuckRegistration(stateDir(), nothingRuns), { kind: 'none' }); + }); + + it('is removed when it names a process that has gone and carries no start time, which agent-device refuses and never removes', () => { + const dir = stateDir({ pid: GONE, version: '0.21.22', port: 50999 }); + assert.deepEqual(clearStuckRegistration(dir, nothingRuns), { kind: 'stuck', pid: GONE }); + assert.equal(existsSync(join(dir, 'daemon.json')), false); + }); + + it('is removed when its start time is blank', () => { + const dir = stateDir({ pid: GONE, processStartTime: ' ' }); + assert.deepEqual(clearStuckRegistration(dir, nothingRuns), { kind: 'stuck', pid: GONE }); + assert.equal(existsSync(join(dir, 'daemon.json')), false); + }); + + it('is left to agent-device when it carries a start time, because agent-device replaces that one by itself', () => { + const registration = { pid: GONE, processStartTime: 'Fri Oct 9 11:59:51 2026' }; + const dir = stateDir(registration); + assert.deepEqual(clearStuckRegistration(dir, nothingRuns), { kind: 'replaceable', pid: GONE }); + assert.deepEqual(JSON.parse(readFileSync(join(dir, 'daemon.json'), 'utf8')), registration); + }); + + it('is left alone while its process runs, with or without a start time', () => { + for (const registration of [{ pid: 4242 }, { pid: 4242, processStartTime: 'Fri Oct 9 11:59:51 2026' }]) { + const dir = stateDir(registration); + assert.deepEqual(clearStuckRegistration(dir, everythingRuns), { kind: 'running', pid: 4242 }); + assert.ok(existsSync(join(dir, 'daemon.json'))); + } + }); + + it('is left alone when it cannot be read or names no process', () => { + for (const registration of ['{ not json', 'null', '[]', '4242', { port: 50999 }, { pid: '4242' }, { pid: 0 }]) { + const dir = stateDir(registration); + assert.deepEqual(clearStuckRegistration(dir, nothingRuns), { kind: 'unreadable' }); + assert.ok(existsSync(join(dir, 'daemon.json'))); + } + }); + + it('asks the process table by default', () => { + assert.deepEqual(readRegistration(stateDir({ pid: process.pid })), { kind: 'running', pid: process.pid }); + assert.deepEqual(readRegistration(stateDir({ pid: GONE })), { kind: 'stuck', pid: GONE }); + }); +}); + +const activity = (seconds: string, what: string): string => ` t = ${seconds.padStart(8)}s ${what}`; +const RUNNER = '2026-10-09 10:22:30.374123-0400 AgentDeviceRunnerUITests-Runner[5344:74148624] '; +const tagged = (tag: string, fields = ''): string => `${RUNNER}AGENT_DEVICE_RUNNER_${tag}${fields}`; +const COMMAND = 'runner-0F3A9C1E-7B2D-4E5F-8A6B-1C2D3E4F5A6B'; +const copyOf = (log: string): string => scrubDriverLog(log).text; + +const WRITTEN_AGAIN_AS_IT_WAS = [ + activity('0.01', 'Start Test at 2026-10-09 10:22:30.374'), + activity('0.01', 'Set Up'), + ' t = nans Interface orientation changed to Portrait', + activity('1.10', 'Find the Keyboard (First Match)'), + activity('1.20', 'Find the Window (Element at index 0)'), + activity('1.30', 'Find the Window at {{0.0, 0.0}, {402.0, 874.0}}'), + activity('1.40', 'Find the TextField (Element at index 3) (retry 1)'), + activity('1.50', 'Checking existence of `Keyboard (First Match)`'), + activity('1.60', 'Get all elements bound by index for: Descendants matching type Alert'), + activity('1.70', 'Find: Descendants matching type TextField'), + activity('1.80', 'Find: Element at index 2'), + activity('1.90', ' Synthesize event'), + activity('2.00', 'Requesting snapshot of accessibility hierarchy for app with pid 5344'), + activity('2.10', 'Collecting debug information to assist test failure triage'), + activity('2.20', 'Wait for com.apple.springboard to idle'), + activity('2.30', 'Ignoring failure to get hierarchy for remote element in process 11410 (Error getting main window kAXErrorInvalidUIElement)'), + activity('2.40', 'Tear Down'), + tagged('WAITING'), + tagged('PORT', '=50999'), + tagged('COMMAND_ACCEPTED', ` command=snapshot commandId=${COMMAND}`), + tagged('COMMAND_COMPLETED', ` command=snapshot commandId=${COMMAND} ok=1`), + tagged('TEXT_ENTRY_PHASE', ` commandId=${COMMAND} phase=type-first durationMs=214 chars=27 mode=append`), + tagged('KEYBOARD_BAND_FACT', ' kind=absent reason=- elapsedMs=3'), + tagged('SYNTHESIZED_DISPATCH', ' kind=tap point=(195.0,520.0) reference=(0.0,0.0,402.0,874.0) orientation=1'), + tagged('SCROLL_VIEWPORT', ' kind=scroll axHealth=healthy keyboardPolicy=requiredWhenAvailable decision=noKeyboard keyboardMinY=none swipeHeight=437.0'), + tagged('SNAPSHOT_RECOVERED', ' backend=private-ax reason=XCTest-backed snapshot tiers were deferred after recent slow accessibility work on this screen'), + tagged('PRIVATE_AX_SNAPSHOT_DEPTH_RETRY', ' depth=24 error=Could not match active AX application for XCTest application'), + tagged('PRIVATE_AX_SNAPSHOT_FAILED', '=Could not match active AX application for XCTest application'), + tagged('WAIT_RESULT', '=XCTWaiterResult(rawValue: 2)'), + `${RUNNER}[DEBUG-1874] synthesize posted 27 chars status=0 tookMs=2383`, + `${RUNNER}[Default] Running tests...`, + `${RUNNER}[connection] Connection interrupted: will attempt to reconnect`, + '2026-10-09 10:24:29.458 xcodebuild[31701:74147400] [MT] IDETestOperationsObserverDebug: 120.521 elapsed -- Testing started completed.', + '2026-10-09 10:24:29.458 xcodebuild[31701:74147400] [MT] IDETestOperationsObserverDebug: 120.521 sec, +120.521 sec -- end', + 'Testing started', + '\t Executed 1 test, with 0 failures (0 unexpected) in 119.894 (119.897) seconds', + '** TEST EXECUTE FAILED **', + 'Failing tests:', + '\tRunnerTests.testCommand()', + 'AGENT_DEVICE_DAEMON_PORT=50999', + 'AGENT_DEVICE_DAEMON_HTTP_PORT=51000', + '', +]; + +const WRITTEN_AGAIN_WITHOUT_ITS_FREE_TEXT: ReadonlyArray = [ + [activity('1.00', 'Find the "clerk.auth.signUp.password" SecureTextField'), activity('1.00', `Find the "${LEFT_OUT}" SecureTextField`)], + [activity('1.00', 'Find the "{"screen":"home","sessionId":"sess_2abc"}" StaticText'), activity('1.00', `Find the "${LEFT_OUT}" StaticText`)], + [activity('1.00', 'Find the "Continue" Button (retry 2)'), activity('1.00', `Find the "${LEFT_OUT}" Button (retry 2)`)], + [activity('1.00', "Find the Application 'com.clerk.E2EHost'"), activity('1.00', `Find the Application '${LEFT_OUT}'`)], + [activity('1.00', 'Checking existence of `"Continue" Button`'), activity('1.00', `Checking existence of \`"${LEFT_OUT}" Button\``)], + [activity('1.00', ' Check for interrupting elements affecting "Continue" Button'), activity('1.00', ` Check for interrupting elements affecting "${LEFT_OUT}" Button`)], + [activity('1.00', 'Tap "Continue" Button[195.0, 520.0] -> (195.0, 520.0)'), activity('1.00', `Tap "${LEFT_OUT}" Button[195.0, 520.0] -> (195.0, 520.0)`)], + [activity('1.00', "Tap Application 'com.clerk.E2EHost'[0.5, 0.6] -> (201.0, 524.4)"), activity('1.00', `Tap Application '${LEFT_OUT}'[0.5, 0.6] -> (201.0, 524.4)`)], + [activity('1.00', `Type 'V' into "clerk.auth.signUp.password" SecureTextField`), activity('1.00', `Type ${LEFT_OUT}`)], + [activity('1.00', `Type 'Verify-r20261009' into Application 'com.clerk.E2EHost'`), activity('1.00', `Type ${LEFT_OUT}`)], + [activity('1.00', `Get all elements bound by index for: Elements matching predicate 'label CONTAINS[c] "424242"'`), activity('1.00', `Get all elements bound by index for: Elements matching predicate ${LEFT_OUT}`)], + [activity('1.00', `Get number of matches for: Elements containing elements matching predicate 'value ==[c] "424242"'`), activity('1.00', `Get number of matches for: Elements containing elements matching predicate ${LEFT_OUT}`)], + [activity('1.00', 'Wait for com.clerk.E2EHost to idle'), activity('1.00', `Wait for ${LEFT_OUT} to idle`)], + [activity('1.00', 'Ignoring failure to wait for app to idle'), activity('1.00', `Ignoring failure to ${LEFT_OUT}`)], + [' t = 1e3s Synthesize event', ` t = ${LEFT_OUT} Synthesize event`], + [tagged('ACTIVATE', ' bundle=com.clerk.E2EHost state=4 reason=bundle_changed'), tagged('ACTIVATE', ` bundle=${LEFT_OUT} state=4 reason=bundle_changed`)], + [tagged('ALERT_ACTIVATION', ' action=dismiss label=Not Now frame=(1.0,2.0,3.0,4.0) point=(5.0,6.0)'), tagged('ALERT_ACTIVATION', ` action=dismiss ${LEFT_OUT}`)], + [tagged('COMMAND_FAILED', ` command=type commandId=${COMMAND} error=Error Domain=AgentDeviceRunner Code=1 "main thread execution timed out"`), tagged('COMMAND_FAILED', ` command=type commandId=${COMMAND} error=${LEFT_OUT}`)], + [tagged('TEXT_INPUT_PROBE_UNAVAILABLE', ' issue=Failed to get matching snapshot: No matches found for Descendants matching type TextField from input {('), tagged('TEXT_INPUT_PROBE_UNAVAILABLE', ` ${LEFT_OUT}`)], + [tagged('LISTENER_FAILED', '=POSIXErrorCode(rawValue: 48): Address already in use'), tagged('LISTENER_FAILED', `=${LEFT_OUT}`)], + [tagged('SNAPSHOT_RECOVERED', ' backend=queries reason=a reason the copy has not read chars=3'), tagged('SNAPSHOT_RECOVERED', ` backend=queries reason=${LEFT_OUT}`)], + [tagged('WAITING', ' and more'), tagged('WAITING', ` ${LEFT_OUT}`)], + [tagged('ACTIVATE', ' state=4 aKeyTheCopyHasNotRead=1 reason=bundle_changed'), tagged('ACTIVATE', ` state=4 ${LEFT_OUT}`)], + [`${RUNNER}[connection] Handshake failed with error: `, `${RUNNER}[connection] Handshake failed with error: ${LEFT_OUT}`], + ["Test Suite 'RunnerTests' started at 2026-10-09 10:22:30.442.", `Test Suite '${LEFT_OUT}' started at 2026-10-09 10:22:30.442.`], + ["Test Case '-[AgentDeviceRunnerUITests.RunnerTests testCommand]' failed (119.894 seconds).", `Test Case '${LEFT_OUT}' failed (119.894 seconds).`], + [ + '/Users/runner/work/RunnerTests+Snapshot.swift:88: error: -[AgentDeviceRunnerUITests.RunnerTests testCommand] : Failed to get matching snapshot: No matches found for Element at index 2 from input {(', + `${LEFT_OUT}: error: -[AgentDeviceRunnerUITests.RunnerTests testCommand] : Failed to get matching snapshot: ${LEFT_OUT}`, + ], + ['\tNo matching device (40FE33C1-55A4-49F7-866C-4B594826A371) in set at /Users/runner/Library/Developer/CoreSimulator/Devices', `\tNo matching device (${LEFT_OUT}) in set at ${LEFT_OUT}`], + ['Daemon error: listen EADDRINUSE: address already in use 127.0.0.1:50999', `Daemon error: ${LEFT_OUT}`], + ['Daemon registration contended; exiting.', `Daemon registration ${LEFT_OUT}; exiting.`], + [ + '{"ts":"2026-10-09T14:22:30.374Z","level":"info","phase":"ios_runner_session_detached","session":"daemon","command":"daemon","data":{"deviceId":"40FE33C1-55A4-49F7-866C-4B594826A371","lane":"simulator","sessionId":"verify-ios-abc-0","runnerPid":5344,"port":50999,"runnerLogPath":"/Users/runner/runner.log"}}', + '2026-10-09T14:22:30.374Z info ios_runner_session_detached lane=simulator runnerPid=5344 port=50999 fieldsLeftOut=5', + ], +]; + +const OF_NO_SHAPE_IT_KNOWS: ReadonlyArray = [ + [activity('3.20', `Press key 'V'`), 'activity'], + [activity('3.20', 'Typing into the focused field'), 'activity'], + [activity('3.20', `Tap "V" Key[20.0, 700.0]`), 'activity'], + [activity('3.20', `Find the 'V' Key`), 'activity'], + [activity('3.20', 'Find the "“V”" Key'), 'activity'], + [activity('3.20', 'Checking existence of `"V" Key`'), 'activity'], + [activity('3.20', 'Get all elements bound by accessibility element for 5123'), 'activity'], + [activity('3.20', 'Interface orientation changed to a word the copy has not read'), 'activity'], + [activity('3.20', 'Find the Keyboard (First Match) and more'), 'activity'], + [activity('3.20', ''), 'activity'], + [`\u001b[2mt = 3.00s Find the Keyboard (First Match)`, 'line'], + [`${activity('1.00', 'Find the Keyboard (First Match)')}${activity('1.10', `Type 'Verify-r20261009' into "password" SecureTextField`)}`, 'activity'], + [`{"phase":"runner_output","data":{"chunk":"${activity('1.00', 'Find the Keyboard (First Match)')}\\n${activity('1.10', `Type 'V' into Application 'com.clerk.E2EHost'`)}\\n"}}`, 'line'], + ['{"ts":"2026-10-09T14:22:30.374Z","level":"info","phase":"request_failed","data":{"message":"text entry verification failed"}}', 'line'], + ['{"ts":"2026-10-09T14:22:30.374Z","level":"info","phase":"ios_runner_session_detached","phase":"a phase the copy has not read","data":{}}', 'line'], + [tagged('A_TAG_THE_RUNNER_DOES_NOT_HAVE'), 'line'], + [tagged('PORTAL', '=50999'), 'line'], + [`2026-10-09 10:22:30.374123-0400 AnotherProcess[5344:74148624] AGENT_DEVICE_RUNNER_WAITING`, 'line'], + [`${RUNNER}[] nw_listener_socket_inbox_create_socket setsockopt SO_NECP_LISTENUUID failed [2: No such file or directory]`, 'line'], + [" TextField, 0x1, {{0.0, 0.0}, {100.0, 44.0}}, identifier: 'code', placeholderValue: 'Code', value: 424242", 'line'], + [' Application, pid: 5344', 'line'], + [' "com.apple.CoreSimulator.SimDeviceType" = 12;', 'line'], + ['Command line invocation:', 'line'], + [' /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun "/Users/runner/x.xctestrun"', 'line'], + ['Testing started\r', 'line'], + [`${activity('1.00', 'Find the "Continue')}\u2028on the next line" Button`, 'line'], + [`${tagged('WAITING')}\u2029`, 'line'], + ['the app said something', 'line'], +]; + +describe('what a copy of a driver log keeps', () => { + it('writes a line again as it was when its whole shape is one the copy knows and no part of it is free text', () => { + for (const line of WRITTEN_AGAIN_AS_IT_WAS) assert.deepEqual(scrubDriverLog(line), { text: line, lines: line === '' ? 0 : 1, whole: line === '' ? 0 : 1, partly: 0, unknownActivities: 0, unknownLines: 0 }, line); + const log = `${WRITTEN_AGAIN_AS_IT_WAS.join('\n')}\n`; + assert.deepEqual(scrubDriverLog(log), { text: log, lines: WRITTEN_AGAIN_AS_IT_WAS.length, whole: WRITTEN_AGAIN_AS_IT_WAS.length, partly: 0, unknownActivities: 0, unknownLines: 0 }); + }); + + it('keeps the fixed words, numbers and times of a line it knows, and leaves out its label, typed text, predicate, name or message', () => { + for (const [line, kept] of WRITTEN_AGAIN_WITHOUT_ITS_FREE_TEXT) assert.deepEqual(scrubDriverLog(line), { text: kept, lines: 1, whole: 0, partly: 1, unknownActivities: 0, unknownLines: 0 }, line); + }); + + it('leaves out a line of any other shape, whole, and counts it', () => { + for (const [line, kind] of OF_NO_SHAPE_IT_KNOWS) { + assert.deepEqual(scrubDriverLog(line), { text: linesLeftOut(1), lines: 1, whole: 0, partly: 0, unknownActivities: kind === 'activity' ? 1 : 0, unknownLines: kind === 'line' ? 1 : 0 }, line); + } + }); + + it('says in the place of each run of lines it left out how many they were', () => { + const known = activity('1.00', 'Synthesize event'); + const unknown = 'the app said something'; + assert.deepEqual(scrubDriverLog([known, unknown, activity('1.10', `Press key 'V'`), unknown, known, '', unknown].join('\n')), { + text: [known, linesLeftOut(3), known, '', linesLeftOut(1)].join('\n'), + lines: 7, + whole: 3, + partly: 0, + unknownActivities: 1, + unknownLines: 3, + }); + assert.equal(linesLeftOut(1), '<1 line left out>'); + assert.equal(linesLeftOut(3), '<3 lines left out>'); + assert.equal(copyOf(`${unknown}\n${unknown}\n`), `${linesLeftOut(2)}\n`); + assert.equal(copyOf(''), ''); + }); + + it('leaves out typed text whatever way the time of the activity is written, and does not keep a line for its time', () => { + for (const time of ['12.10s', 'nans', 'infs', '-1.5s', '1e3s', '12,5s', '12.1ms', 'soon', "'Q'"]) { + for (const spaces of [' ', '', ' ']) { + const typed = scrubDriverLog(` t =${spaces}${time} Type 'Q' into "pw" SecureTextField`); + assert.equal(typed.text.includes("'Q'"), false, `t =${spaces}${time}`); + assert.equal(typed.text.includes('pw'), false, `t =${spaces}${time}`); + assert.equal(copyOf(` t =${spaces}${time} Press key 'Q'`), linesLeftOut(1), `t =${spaces}${time}`); + assert.equal(copyOf(` t =${spaces}${time} Synthesize event`), ` t =${spaces}${/^(?:[\d.-]+|nan)s$/.test(time) ? time : LEFT_OUT} Synthesize event`); + } + } + }); + + const CANARY = 'ZqxKpmWvJ'; + const holdsTheCanary = (text: string): boolean => text.toLowerCase().includes(CANARY.toLowerCase()); + + it('keeps no text of a caller from the lines the security review probed', () => { + const typed = `Type '${CANARY}' into "pw" SecureTextField`; + const probes = [ + ...['nans', 'infs', '-1.5s', '1e3s', '12,5s', '12.1ms'].map((time) => ` t = ${time} ${typed}`), + ` t =12.1s ${typed}`, + activity('1.00', `Find the Elements matching predicate 'label CONTAINS[c] "${CANARY}" OR value ==[c] "${CANARY}"'`), + activity('1.00', `Checking existence of \`Elements matching predicate 'value ==[c] "${CANARY}"'\``), + activity('1.00', `Get number of matches for: Elements containing elements matching predicate 'label CONTAINS[c] "${CANARY}"'`), + activity('1.00', `Wait for ${CANARY} to idle`), + activity('1.00', `Ignoring failure to read value ${CANARY}`), + activity('1.00', `Find the "${CANARY}" StaticText`), + tagged('ELEMENT_TAP_IGNORED_EXCEPTION', `=no element with value ${CANARY}`), + tagged('LISTENER_FAILED', `=${CANARY}`), + tagged('SEND_FAILED', `=${CANARY}`), + tagged('PRIVATE_AX_SNAPSHOT_FAILED', `=${CANARY}`), + `${tagged('TEXT_INPUT_PROBE_UNAVAILABLE', ' issue=Failed to get matching snapshot: No matches found for Descendants matching type TextField from input {(')}\n TextField, 0x1, {{0.0, 0.0}, {100.0, 44.0}}, identifier: 'code', value: ${CANARY}\n)}`, + ` TextField, 0x1, {{0.0, 0.0}, {100.0, 44.0}}, identifier: 'code', placeholderValue: 'Code', value: ${CANARY}`, + tagged(CANARY.toUpperCase()), + tagged('ALERT_ACTIVATION', ` action=dismiss label=${CANARY} frame=(1,2,3,4) point=(5,6)`), + tagged('ACTIVATE', ` bundle=${CANARY} state=4 reason=${CANARY}`), + tagged('ACTIVATE', ` ${CANARY}=1`), + tagged('ACTIVATE', ` state=4 ${CANARY}=1 reason=bundle_changed`), + tagged('KEYBOARD_BAND_FACT', ` kind=absent reason=- ${CANARY}=${CANARY}`), + `the app said ${CANARY}`, + activity('1.00', `Press key '${CANARY}'`), + activity('1.00', `Type key '${CANARY}'`), + activity('1.00', `Set value of "pw" SecureTextField to '${CANARY}'`), + activity('1.00', `Paste '${CANARY}' into "pw" SecureTextField`), + activity('1.00', `Tap «${CANARY}» Key[20.0, 700.0]`), + `${activity('1.00', 'Find the Keyboard (First Match)')}${activity('1.10', typed)}`, + `{"phase":"runner_output","data":{"chunk":"${activity('1.00', 'Find the Keyboard')}\\n${activity('1.10', `Type \\"${CANARY}\\" into Application 'com.clerk.E2EHost'`)}\\n"}}`, + `\u001b[2mt = 3.00s Find the "${CANARY}" Button`, + `{"ts":"2026-10-09T14:22:30.374Z","level":"info","phase":"ios_runner_session_detached","session":"${CANARY}","data":{"lane":"${CANARY}","reason":"${CANARY}","port":"${CANARY}","${CANARY}":1}}`, + `Daemon error: ${CANARY}`, + `Daemon registration ${CANARY}; exiting.`, + `Test Case '${CANARY}' started.`, + ]; + for (const probe of probes) assert.equal(holdsTheCanary(copyOf(probe)), false, probe); + }); + + it('puts no text into a copy that stands in a line where the shape has none, wherever in the line it stands', () => { + let tried = 0; + for (const line of [...WRITTEN_AGAIN_AS_IT_WAS, ...WRITTEN_AGAIN_WITHOUT_ITS_FREE_TEXT.map(([written]) => written)]) { + for (let at = 0; at <= line.length; at += 1) { + for (const text of [CANARY, ` ${CANARY} `, `"${CANARY}"`, `'${CANARY}'`, `=${CANARY}`, `${CANARY}=`, ` ${CANARY}=1`, ` ${CANARY}=${CANARY} `, `${CANARY}\r`]) { + tried += 1; + const withText = `${line.slice(0, at)}${text}${line.slice(at)}`; + assert.equal(holdsTheCanary(copyOf(withText)), false, withText); + } + } + } + assert.ok(tried > 40_000, `${tried} lines tried`); + }); + + it('redacts a value the CLI used where a shape lets its characters through, which only digits can be', () => { + const digits = `90817${randomBytes(4).readUInt32BE(0)}`.padEnd(12, '7'); + protect(digits); + assert.equal(copyOf(tagged('PORT', `=${digits}`)), tagged('PORT', '=')); + assert.equal(copyOf(activity('1.00', `Find the Window (Element at index ${digits})`)), activity('1.00', 'Find the Window (Element at index )')); + }); + + it('reads four megabytes of lines made to match slowly in less than three seconds each', { timeout: 60_000 }, () => { + const size = 4 * 1024 * 1024; + const slow = { + 'the ends of many labels in one line': activity('1.00', `Find the "${'" Button '.repeat(size / 9)}`), + 'the starts of many frames in one line': activity('1.00', `Find the "${'" Window at {{1111111111111111, '.repeat(size / 34)}`), + 'the middles of many failures in one line': ': error: -[AgentDeviceRunnerUITests.RunnerTests testCommand] : '.repeat(size / 64), + 'many fields in one line': tagged('ACTIVATE', ' state=4'.repeat(size / 8)), + 'many short activities': `${activity('1.00', 'Find the "a" Button')}\n`.repeat(size / 36), + 'many starts of a daemon event': '{"ts":"2026-10-09T\n'.repeat(size / 19), + 'spaces after a time and a carriage return': `${activity('1.00', ' '.repeat(110_000))}x\r`, + 'the middles of many failures and a carriage return': `${'a: error: -[AgentDeviceRunnerUITests.RunnerTests testCommand] : '.repeat(11_000)}\r`, + 'the middles of many missing devices and a line separator': `\tNo matching device (${') in set at '.repeat(27_000)}\u2028`, + 'many lines of one character': '{\n'.repeat(size / 2), + }; + for (const [name, log] of Object.entries(slow)) { + const started = Date.now(); + scrubDriverLog(log); + assert.ok(Date.now() - started < 3_000, `${name}: ${Date.now() - started} ms`); + } + }); +}); + +describe('the driver logs a run keeps', () => { + const GONE_TOO = 2147483645; + const world = (registration?: unknown) => { + const state = stateDir(registration); + const run = mkdtempSync(join(tmpdir(), 'verify-driver-run-')) as EvidencePath; + const kept = (): string[] => (existsSync(join(run, 'driver')) ? readdirSync(join(run, 'driver')).sort() : []); + const read = (name: string): string => readFileSync(join(run, 'driver', name), 'utf8'); + const summary = (): string[] => read('summary.txt').trim().split('\n'); + const runnerLog = (session: string): string => { + mkdirSync(join(state, 'sessions', session), { recursive: true }); + return join(state, 'sessions', session, 'runner.log'); + }; + return { state, run, kept, read, summary, runnerLog, daemonLog: join(state, 'daemon.log') }; + }; + const quiet = (): void => undefined; + const PORT = 'AGENT_DEVICE_DAEMON_PORT=50999\n'; + const NEXT_PORT = 'AGENT_DEVICE_DAEMON_PORT=51000\n'; + const WAITING = `${tagged('WAITING')}\n`; + const READY = `${tagged('LISTENER_READY')}\n`; + const lines = (read: number, whole: number, partly = 0, activities = 0, others = 0): string => + `Of the ${read} line${read === 1 ? '' : 's'} read, the copy keeps ${whole} as written and ${partly} with a label, typed text, name or message left out, and it leaves out ${activities + others} whose shape it does not know: ${activities} with an XCTest activity and ${others} without.`; + const LAST_WORDS = [ + 'These are logs of the agent-device daemon on this machine. A remote session drives its device with a daemon on the runner, and that daemon writes its logs there.', + 'A copy is not the whole log. It holds a line only when the CLI knows the whole shape of the line, and it writes that line again from the fixed words of the shape, its numbers and its times. Where the log shows a label, typed text, a predicate, a name or a message, the copy shows . Lines of any other shape are counted above and shown as one line such as <2 lines left out>. Each copy then passed through the redaction the CLI applies to app.log.', + ]; + + it('says plainly that there is nothing when no daemon wrote a log', () => { + const w = world(); + watchDriver(w.state, w.run, nothingRuns).collect(); + assert.deepEqual(w.kept(), ['summary.txt']); + assert.deepEqual(w.summary(), [ + 'At the start of the run there was no daemon.json, so no agent-device daemon was registered.', + 'At the end of the run there was no daemon.json, so no agent-device daemon was registered.', + 'There is no daemon.log, so no agent-device daemon on this machine wrote one.', + 'No session has a runner.log. The XCTest runner writes one, on iOS only.', + ...LAST_WORDS, + ]); + }); + + it('keeps only what each log gained during the run', () => { + const w = world({ pid: 4242, processStartTime: 'Fri Oct 9 11:59:51 2026' }); + writeFileSync(w.daemonLog, PORT); + writeFileSync(w.runnerLog('verify-ios-abc-0'), WAITING); + writeFileSync(w.runnerLog('verify-ios-abc-screen'), WAITING); + const driver = watchDriver(w.state, w.run, everythingRuns); + appendFileSync(w.daemonLog, NEXT_PORT); + appendFileSync(w.runnerLog('verify-ios-abc-0'), READY); + driver.collect(); + assert.deepEqual(w.kept(), ['daemon.log', 'runner-verify-ios-abc-0.log', 'summary.txt']); + assert.equal(w.read('daemon.log'), NEXT_PORT); + assert.equal(w.read('runner-verify-ios-abc-0.log'), READY); + assert.deepEqual(w.summary(), [ + 'At the start of the run daemon.json named pid 4242, which was running.', + 'At the end of the run daemon.json named pid 4242, which was running.', + `daemon.log is the part of daemon.log that was written during this run: ${NEXT_PORT.length} bytes. ${lines(1, 1)}`, + `runner-verify-ios-abc-0.log is the part of sessions/verify-ios-abc-0/runner.log that was written during this run: ${READY.length} bytes. ${lines(1, 1)}`, + 'sessions/verify-ios-abc-screen/runner.log did not grow during the run, so there is no runner-verify-ios-abc-screen.log.', + ...LAST_WORDS, + ]); + }); + + it('says for each copy how many lines it read, kept as written, kept in part and left out, so that a reader knows the copy is not the log', () => { + const w = world(); + const driver = watchDriver(w.state, w.run, nothingRuns); + writeFileSync(w.daemonLog, `${PORT}Daemon error: the request for /v1/sessions was refused\nsomething no daemon is known to write\n`); + writeFileSync( + w.runnerLog('verify-ios-abc-0'), + [WAITING.trimEnd(), activity('1.00', 'Find the "Continue" Button'), activity('1.10', `Type 'V' into "password" SecureTextField`), activity('1.20', `Press key 'V'`), ' Application, pid: 5344', 'Command line invocation:', ''].join('\n'), + ); + driver.collect(); + assert.equal(w.read('daemon.log'), `${PORT}Daemon error: ${LEFT_OUT}\n${linesLeftOut(1)}\n`); + assert.equal(w.read('runner-verify-ios-abc-0.log'), [WAITING.trimEnd(), activity('1.00', `Find the "${LEFT_OUT}" Button`), activity('1.10', `Type ${LEFT_OUT}`), linesLeftOut(3), ''].join('\n')); + assert.match(w.summary()[2]!, /^daemon\.log is the whole of daemon\.log, which was written or rewritten during this run: \d+ bytes\. Of the 3 lines read, the copy keeps 1 as written and 1 with a label, typed text, name or message left out, and it leaves out 1 whose shape it does not know: 0 with an XCTest activity and 1 without\.$/); + assert.ok(w.summary()[3]!.endsWith(` bytes. ${lines(6, 1, 2, 1, 2)}`), w.summary()[3]); + }); + + it('keeps the whole of a log that a daemon started during the run rewrote, and of a log that did not exist before', () => { + const w = world(); + writeFileSync(w.daemonLog, `${PORT}AGENT_DEVICE_DAEMON_HTTP_PORT=50998\n`); + const driver = watchDriver(w.state, w.run, nothingRuns); + const rewritten = `${NEXT_PORT}AGENT_DEVICE_DAEMON_HTTP_PORT=51001\nDaemon registration contended; exiting.\n`; + writeFileSync(w.daemonLog, rewritten); + writeFileSync(w.runnerLog('verify-ios-abc-0'), WAITING); + driver.collect(); + assert.equal(w.read('daemon.log'), `${NEXT_PORT}AGENT_DEVICE_DAEMON_HTTP_PORT=51001\nDaemon registration ${LEFT_OUT}; exiting.\n`); + assert.equal(w.read('runner-verify-ios-abc-0.log'), WAITING); + assert.deepEqual(w.summary().slice(2, 4), [ + `daemon.log is the whole of daemon.log, which was written or rewritten during this run: ${rewritten.length} bytes. ${lines(3, 2, 1)}`, + `runner-verify-ios-abc-0.log is the whole of sessions/verify-ios-abc-0/runner.log, which was written or rewritten during this run: ${WAITING.length} bytes. ${lines(1, 1)}`, + ]); + }); + + it('keeps the end of a part that is over the limit, from the start of a line, and says how much it left out', () => { + const w = world(); + const whole = `${tagged('ACTIVATE', ' state=4'.repeat(110)).padEnd(1023, '4')}\n`; + assert.equal(whole.length, 1024); + const driver = watchDriver(w.state, w.run, nothingRuns); + writeFileSync(w.runnerLog('verify-ios-abc-0'), `${WAITING}${whole.repeat(DRIVER_LOG_LIMIT_BYTES / 1024)}`); + driver.collect(); + const kept = w.read('runner-verify-ios-abc-0.log'); + assert.equal(kept, whole.repeat(DRIVER_LOG_LIMIT_BYTES / 1024 - 1)); + assert.equal( + w.summary()[3], + `runner-verify-ios-abc-0.log is the whole of sessions/verify-ios-abc-0/runner.log, which was written or rewritten during this run: ${DRIVER_LOG_LIMIT_BYTES + WAITING.length} bytes, of which the first ${1024 + WAITING.length} were left out, because a copy starts at the start of a line and holds ${DRIVER_LOG_LIMIT_BYTES} bytes at most. ${lines(DRIVER_LOG_LIMIT_BYTES / 1024 - 1, DRIVER_LOG_LIMIT_BYTES / 1024 - 1)}`, + ); + }); + + it('starts a part at the start of a line when the log ended in the middle of one as the run began, so no half of a line is kept', () => { + const w = world(); + const half = activity('1.00', 'Find the Window (Element at '); + writeFileSync(w.runnerLog('verify-ios-abc-0'), `${WAITING}${half}`); + const driver = watchDriver(w.state, w.run, nothingRuns); + appendFileSync(w.runnerLog('verify-ios-abc-0'), `index 0)\n${READY}`); + driver.collect(); + assert.equal(w.read('runner-verify-ios-abc-0.log'), READY); + assert.ok(w.summary()[3]!.includes(`: ${'index 0)\n'.length + READY.length} bytes, of which the first ${'index 0)\n'.length} were left out, because a copy starts at the start of a line`), w.summary()[3]); + }); + + it('keeps nothing of a part that is one unfinished line', () => { + const w = world(); + writeFileSync(w.runnerLog('verify-ios-abc-0'), 'AGENT_DEVICE_DAEMON_'); + const driver = watchDriver(w.state, w.run, nothingRuns); + appendFileSync(w.runnerLog('verify-ios-abc-0'), 'PORT=50999'); + driver.collect(); + assert.equal(w.read('runner-verify-ios-abc-0.log'), ''); + }); + + it('writes the copies through the scrub', () => { + const w = world(); + const driver = watchDriver(w.state, w.run, nothingRuns); + const typed = `${activity('1.00', `Type 'V' into Application 'com.clerk.E2EHost'`)}\n`; + writeFileSync(w.daemonLog, typed); + writeFileSync(w.runnerLog('verify-ios-abc-0'), typed); + driver.collect(); + assert.equal(w.read('daemon.log'), `${activity('1.00', `Type ${LEFT_OUT}`)}\n`); + assert.equal(w.read('runner-verify-ios-abc-0.log'), `${activity('1.00', `Type ${LEFT_OUT}`)}\n`); + }); + + it('copies nothing of a log when the scrub throws, and says so', () => { + const w = world({ pid: GONE }); + writeFileSync(w.daemonLog, PORT); + writeFileSync(w.runnerLog('verify-ios-abc-0'), `${activity('1.00', `Type 'V' into "password" SecureTextField`)}\n`); + const driver = watchDriver(w.state, w.run, nothingRuns, () => { + throw new RangeError('Maximum call stack size exceeded'); + }); + driver.startClean(quiet); + writeFileSync(w.daemonLog, NEXT_PORT); + driver.collect(); + assert.deepEqual(w.kept(), ['summary.txt']); + assert.equal(w.summary()[1], 'daemon.log could not be copied (RangeError), so there is no daemon-that-did-not-start.log.'); + assert.equal(w.summary()[4], 'daemon.log could not be copied (RangeError), so there is no daemon.log.'); + assert.equal(w.summary()[5], 'sessions/verify-ios-abc-0/runner.log could not be copied (RangeError), so there is no runner-verify-ios-abc-0.log.'); + }); + + it('copies nothing of a log whose size or start it could not read as the run began, because it cannot tell which part the run wrote', () => { + const w = world(); + mkdirSync(w.daemonLog); + mkdirSync(w.runnerLog('verify-ios-abc-0')); + const driver = watchDriver(w.state, w.run, nothingRuns); + rmSync(w.daemonLog, { recursive: true }); + rmSync(w.runnerLog('verify-ios-abc-0'), { recursive: true }); + writeFileSync(w.daemonLog, `${PORT}${NEXT_PORT}`); + writeFileSync(w.runnerLog('verify-ios-abc-0'), `${WAITING}${READY}`); + writeFileSync(w.runnerLog('verify-ios-abc-1'), WAITING); + driver.collect(); + assert.deepEqual(w.kept(), ['runner-verify-ios-abc-1.log', 'summary.txt']); + assert.deepEqual(w.summary().slice(2, 5), [ + 'daemon.log could not be read when the run began, so the CLI cannot tell which part of it this run wrote, and there is no daemon.log.', + 'sessions/verify-ios-abc-0/runner.log could not be read when the run began, so the CLI cannot tell which part of it this run wrote, and there is no runner-verify-ios-abc-0.log.', + `runner-verify-ios-abc-1.log is the whole of sessions/verify-ios-abc-1/runner.log, which was written or rewritten during this run: ${WAITING.length} bytes. ${lines(1, 1)}`, + ]); + }); + + it('copies no runner.log when it could not list the sessions as the run began, or as it ended', () => { + for (const unreadable of ['as the run began', 'as the run ended'] as const) { + const w = world(); + const sessions = join(w.state, 'sessions'); + if (unreadable === 'as the run began') writeFileSync(sessions, ''); + const driver = watchDriver(w.state, w.run, nothingRuns); + rmSync(sessions, { recursive: true, force: true }); + if (unreadable === 'as the run began') writeFileSync(w.runnerLog('verify-ios-abc-0'), WAITING); + else writeFileSync(sessions, ''); + driver.collect(); + assert.deepEqual(w.kept(), ['summary.txt']); + assert.equal( + w.summary()[3], + `The sessions of agent-device could not be read ${unreadable === 'as the run began' ? 'when the run began' : 'when the run ended'}, so the CLI cannot tell which part of a runner.log this run wrote, and no runner.log is copied.`, + ); + } + }); + + it('copies no runner.log of a session whose name is not one the CLI gives, and does not write the name', () => { + const w = world(); + const driver = watchDriver(w.state, w.run, nothingRuns); + for (const session of ['a b', 'a\nforged line', 'x'.repeat(65), 'x'.repeat(250), 'verify-ios-abc-0']) writeFileSync(w.runnerLog(session), WAITING); + driver.collect(); + assert.deepEqual(w.kept(), ['runner-verify-ios-abc-0.log', 'summary.txt']); + assert.deepEqual(w.summary().slice(3, 5), [ + 'The runner.log of 4 sessions is not copied, because the name of the session is not one the CLI gives: letters, digits, dots, dashes and underscores, 64 at most.', + `runner-verify-ios-abc-0.log is the whole of sessions/verify-ios-abc-0/runner.log, which was written or rewritten during this run: ${WAITING.length} bytes. ${lines(1, 1)}`, + ]); + assert.equal(w.summary().length, 7); + }); + + it('redacts a value the CLI used in the summary and in the name of a copy, where the name of a session can put it', () => { + const known = `known-${randomBytes(12).toString('hex')}`; + protect(known); + const w = world(); + const driver = watchDriver(w.state, w.run, nothingRuns); + writeFileSync(w.runnerLog(known), WAITING); + mkdirSync(join(w.runnerLog(`${known}-too`))); + driver.collect(); + assert.deepEqual(w.kept(), ['runner--redacted-.log', 'summary.txt']); + assert.equal(w.read('summary.txt').includes(known), false); + assert.deepEqual(w.summary().slice(3, 5), [ + `runner--redacted-.log is the whole of sessions//runner.log, which was written or rewritten during this run: ${WAITING.length} bytes. ${lines(1, 1)}`, + 'sessions/-too/runner.log could not be copied (EISDIR), so there is no runner--redacted--too.log.', + ]); + }); + + it('removes the registration of a daemon that failed to start, keeps that daemon\'s log as it found it, and says what it did', () => { + const w = world({ pid: GONE }); + const lastWords = `${PORT}Daemon error: listen EADDRINUSE: address already in use 127.0.0.1:50999\n`; + writeFileSync(w.daemonLog, lastWords); + const said: string[] = []; + const driver = watchDriver(w.state, w.run, nothingRuns); + driver.startClean((line) => said.push(line)); + assert.equal(existsSync(join(w.state, 'daemon.json')), false); + assert.deepEqual(said, [`driver removed the agent-device registration of pid ${GONE}, which is not running and has no start time; a daemon failed to start, and agent-device would refuse every command until the file was gone`]); + writeFileSync(w.daemonLog, NEXT_PORT); + driver.collect(); + assert.deepEqual(w.kept(), ['daemon-that-did-not-start.log', 'daemon.log', 'summary.txt']); + assert.equal(w.read('daemon-that-did-not-start.log'), `${PORT}Daemon error: ${LEFT_OUT}\n`); + assert.deepEqual(w.summary().slice(0, 5), [ + `At the start of the run daemon.json named pid ${GONE}, which was not running, and it carried no start time, which is what a daemon that fails to start leaves; agent-device refuses that registration and never removes it.`, + `daemon-that-did-not-start.log is the whole of daemon.log as it was when the CLI removed that registration: ${lastWords.length} bytes. ${lines(2, 1, 1)}`, + `Before a group of tests started, the CLI removed the daemon.json that named pid ${GONE}, which was not running, so that e2e could start a new daemon. The CLI does this once in a run.`, + 'At the end of the run there was no daemon.json, so no agent-device daemon was registered.', + `daemon.log is the whole of daemon.log, which was written or rewritten during this run: ${NEXT_PORT.length} bytes. ${lines(1, 1)}`, + ]); + }); + + it('says that the daemon that failed to start left no log, when it left none', () => { + const w = world({ pid: GONE }); + const driver = watchDriver(w.state, w.run, nothingRuns); + driver.startClean(quiet); + driver.collect(); + assert.deepEqual(w.kept(), ['summary.txt']); + assert.equal(w.summary()[1], 'The daemon that failed to start left no daemon.log.'); + }); + + it('removes a registration once in a run, and names both daemons when the second fails to start too', () => { + const w = world({ pid: GONE }); + const said: string[] = []; + const driver = watchDriver(w.state, w.run, nothingRuns); + assert.equal(driver.daemonThatDidNotStart(), `agent-device's daemon did not start: it left a registration for pid ${GONE}, which is not running`); + driver.startClean((line) => said.push(line)); + assert.equal(driver.daemonThatDidNotStart(), null); + writeFileSync(join(w.state, 'daemon.json'), JSON.stringify({ pid: GONE_TOO })); + assert.equal( + driver.daemonThatDidNotStart(), + `agent-device's daemon did not start: it left a registration for pid ${GONE_TOO}, which is not running, as the daemon before it had (pid ${GONE}), whose registration the CLI removed once in this run and does not remove a second time`, + ); + driver.startClean((line) => said.push(line)); + assert.equal(said.length, 1); + assert.deepEqual(JSON.parse(readFileSync(join(w.state, 'daemon.json'), 'utf8')), { pid: GONE_TOO }); + driver.collect(); + assert.equal(w.summary().at(-5), `At the end of the run daemon.json named pid ${GONE_TOO}, which was not running, and it carried no start time, which is what a daemon that fails to start leaves; agent-device refuses that registration and never removes it.`); + }); + + it('gives a later group a clean start when the daemon fails to start during the run, and keeps what that daemon logged', () => { + const w = world(); + const driver = watchDriver(w.state, w.run, nothingRuns); + driver.startClean(quiet); + writeFileSync(join(w.state, 'daemon.json'), JSON.stringify({ pid: GONE })); + writeFileSync(w.daemonLog, PORT); + driver.startClean(quiet); + assert.equal(existsSync(join(w.state, 'daemon.json')), false); + writeFileSync(w.daemonLog, NEXT_PORT); + driver.collect(); + assert.equal(w.read('daemon-that-did-not-start.log'), PORT); + assert.equal(w.read('daemon.log'), NEXT_PORT); + }); + + it('leaves a registration alone that agent-device can use or replace', () => { + for (const [registration, alive] of [[{ pid: 4242 }, everythingRuns], [{ pid: GONE, processStartTime: 'Fri Oct 9 11:59:51 2026' }, nothingRuns]] as const) { + const w = world(registration); + const driver = watchDriver(w.state, w.run, alive); + driver.startClean(() => assert.fail('nothing to say')); + assert.ok(existsSync(join(w.state, 'daemon.json'))); + assert.equal(driver.daemonThatDidNotStart(), null); + } + }); + + it('still writes the summary when a log cannot be read', () => { + const w = world(); + writeFileSync(join(w.state, 'daemon.json'), JSON.stringify({ pid: GONE })); + const driver = watchDriver(w.state, w.run, nothingRuns); + mkdirSync(w.daemonLog); + mkdirSync(w.runnerLog('verify-ios-abc-0')); + driver.startClean(quiet); + driver.collect(); + assert.deepEqual(w.kept(), ['summary.txt']); + assert.equal(w.summary()[1], 'daemon.log could not be copied (EISDIR), so there is no daemon-that-did-not-start.log.'); + assert.equal(w.summary()[4], 'daemon.log could not be copied (EISDIR), so there is no daemon.log.'); + assert.equal(w.summary()[5], 'sessions/verify-ios-abc-0/runner.log could not be copied (EISDIR), so there is no runner-verify-ios-abc-0.log.'); + assert.deepEqual(w.summary().slice(-2), LAST_WORDS); + }); +}); diff --git a/integration/expo-native/test/e2e.test.ts b/integration/expo-native/test/e2e.test.ts new file mode 100644 index 00000000000..33c0db3cf85 --- /dev/null +++ b/integration/expo-native/test/e2e.test.ts @@ -0,0 +1,171 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { collectScreenshots, e2eOutputDir, parseE2EReport, planE2E, resolveSpecs } from '../src/core/e2e.ts'; +import { manifestDrift } from '../src/core/manifest.ts'; +import type { EvidencePath, SpecRef } from '../src/core/types.ts'; +import { SAMPLE_INPUTS } from '../testing/sample-inputs.ts'; + +function packageDir(): string { + const dir = mkdtempSync(join(tmpdir(), 'verify-specs-')); + for (const file of ['specs/golden/auth-start/opens.e2e.ts', 'specs/golden/sign-up/request-code.e2e.ts', 'specs/explored/probe.e2e.ts', 'specs/fixtures.ts']) { + mkdirSync(join(dir, file, '..'), { recursive: true }); + writeFileSync(join(dir, file), ''); + } + return dir; +} + +describe('resolveSpecs', () => { + it('expands features, feature/spec, paths, and --all', () => { + const dir = packageDir(); + assert.deepEqual(resolveSpecs(dir, { selectors: ['auth-start'] }), [{ kind: 'golden', path: 'specs/golden/auth-start/opens.e2e.ts', feature: 'auth-start' }]); + assert.deepEqual(resolveSpecs(dir, { selectors: ['sign-up/request-code'] }).map((s) => s.path), ['specs/golden/sign-up/request-code.e2e.ts']); + assert.deepEqual(resolveSpecs(dir, { selectors: ['specs/explored/probe.e2e.ts'] }, dir), [{ kind: 'explored', path: 'specs/explored/probe.e2e.ts', feature: null }]); + assert.deepEqual(resolveSpecs(dir, { all: true }).map((s) => s.path), ['specs/golden/auth-start/opens.e2e.ts', 'specs/golden/sign-up/request-code.e2e.ts']); + }); + + it('lists every feature for an unknown selector', () => { + assert.throws(() => resolveSpecs(packageDir(), { selectors: ['profile'] }), { code: 'NO_SPECS', message: 'no specs match profile', fix: /name a feature \(auth-start, sign-up\)/ }); + }); +}); + +describe('planE2E', () => { + it('passes selection flags through and points output inside the run', () => { + const plan = planE2E(SAMPLE_INPUTS, [{ kind: 'golden', path: 'specs/golden/a/b.e2e.ts', feature: null }], { verb: 'run', selection: { all: true }, grep: 'x', video: true, retries: 1, githubReport: false, waitSeconds: 0 }, '/package', e2eOutputDir('/package/.verify/runs/r20261002-141210-7c1e' as EvidencePath, 0)); + assert.deepEqual(plan.args, [ + 'run', 'specs/golden/a/b.e2e.ts', '--config', 'e2e.config.ts', '--target', 'ios', + '--output', '.verify/runs/r20261002-141210-7c1e/e2e', '--reporter', 'list,markdown,junit', '--retries', '1', + '--grep', 'x', '--pass-with-no-tests', + ]); + assert.deepEqual(plan.env, { + CLERK_E2E_PLATFORM: 'ios', + CLERK_E2E_DEVICE: 'FDF0CD9E-CF9E-42B6-AE3A-116A665F7EF3', + CLERK_E2E_DEVICE_SESSION: 'verify-ios-abc', + CLERK_PUBLISHABLE_KEY: 'pk_test_ZXhhbXBsZS5jbGVyay5hY2NvdW50cy5kZXYk', + CLERK_E2E_API_URL: 'http://127.0.0.1:4010/v1', + CLERK_E2E_API_TOKEN_FILE: '/package/.verify/scratch/r/broker-token', + CLERK_E2E_RUN_ID: 'r20261002-141210-7c1e', + AGENT_DEVICE_STATE_DIR: '/package/.verify/agent-device', + E2E_TELEMETRY_DISABLED: '1', + }); + const later = planE2E(SAMPLE_INPUTS, [], { verb: 'run', selection: { all: true }, video: true, retries: 0, githubReport: false, waitSeconds: 0 }, '/package', e2eOutputDir('/package/.verify/runs/r20261002-141210-7c1e' as EvidencePath, 2)); + assert.equal(later.args[later.args.indexOf('--output') + 1], '.verify/runs/r20261002-141210-7c1e/e2e-3', 'a later group of the run writes beside the first'); + assert.equal(later.args[later.args.indexOf('--retries') + 1], '0', 'e2e retries once by default when CI is set, so the run always says how many it wants'); + assert.equal(later.args.includes('--pass-with-no-tests'), true, 'every invocation is one group of a run, and a group with nothing left to run must not fail it'); + }); +}); + +describe('parseE2EReport', () => { + const report = { + schemaVersion: 'report-1', + run: { + results: [ + { + id: 'aaaaaaaa11', kind: 'test', titlePath: ['opens'], file: 'specs/golden/auth-start/opens.e2e.ts', platform: 'ios', tags: [], status: 'passed', + attempts: [{ status: 'passed', durationMs: 9100, artifacts: [{ kind: 'screenshot', path: 'ios/x/attempt-0/screenshots/001-auth.png', producer: { kind: 'step', stepId: 's1' } }], steps: [{ id: 's1', api: 'app.screenshot', label: 'auth' }] }], + }, + { + id: 'bbbbbbbb22', kind: 'test', titlePath: ['completes'], file: 'specs/golden/sign-up/complete.e2e.ts', platform: 'ios', tags: [], status: 'skipped', + skip: { cause: 'filtered', reason: 'title does not match --grep' }, attempts: [], + }, + { + id: 'ffffffff66', kind: 'test', titlePath: ['ios-only screen'], file: 'specs/golden/sign-up/request-code.e2e.ts', platform: 'android', tags: [], status: 'skipped', + skip: { cause: 'platform-unavailable', reason: 'test declares platforms [ios]' }, attempts: [], + }, + { + id: 'dddddddd44', kind: 'test', titlePath: ['never ran'], file: 'specs/golden/auth-start/opens.e2e.ts', platform: 'ios', tags: [], status: 'skipped', + skip: { cause: 'infrastructure-unavailable', reason: 'the device could not be opened' }, attempts: [], + }, + { + id: 'cccccccc33', kind: 'test', titlePath: ['fails'], file: 'specs/explored/probe.e2e.ts', platform: 'ios', status: 'timed-out', + attempts: [{ status: 'timed-out', durationMs: 5000, error: { message: 'expect.toBeVisible failed\nobserved: no node' }, failure: { screen: 'art-1', screenshot: 'art-2' }, artifacts: [{ id: 'art-1', kind: 'other', path: 'ios/p/attempt-0/screen.txt' }, { id: 'art-2', kind: 'screenshot', path: 'ios/p/attempt-0/screenshots/001-failure.png' }] }], + }, + ], + }, + }; + + it('keeps the failed attempt of a test that passed on a retry: its error, its failure page, and its screenshot', () => { + const dir = e2eOutputDir(mkdtempSync(join(tmpdir(), 'verify-report-')) as EvidencePath, 0); + mkdirSync(join(dir, 'failures'), { recursive: true }); + writeFileSync(join(dir, 'failures', 'specs_golden_auth-start_opens-opens-eeeeeeee.md'), ''); + mkdirSync(join(dir, 'artifacts', 'ios/o/attempt-0'), { recursive: true }); + writeFileSync(join(dir, 'artifacts', 'ios/o/attempt-0/screen.txt'), ''); + const attempt = (path: string) => ({ failure: { screen: 'screen', screenshot: 'shot' }, artifacts: [{ id: 'screen', kind: 'other', path: `${path}/screen.txt` }, { id: 'shot', kind: 'screenshot', path: `${path}/screenshots/001-failure.png` }] }); + const retried = (status: string, second: object) => ({ + schemaVersion: 'report-1', + run: { + results: [ + { + id: 'eeeeeeee55', kind: 'test', titlePath: ['opens'], file: 'specs/golden/auth-start/opens.e2e.ts', platform: 'ios', tags: [], status, + attempts: [{ status: 'failed', durationMs: 4000, error: { message: 'tap failed\nthe runner session ended' }, ...attempt('ios/o/attempt-0') }, { durationMs: 6000, ...second }], + }, + ], + }, + }); + + const opens: readonly SpecRef[] = [{ kind: 'golden', path: 'specs/golden/auth-start/opens.e2e.ts', feature: null }]; + const [flaky] = parseE2EReport(retried('flaky', { status: 'passed' }), opens, dir); + assert.equal(flaky!.status, 'flaky'); + assert.equal(flaky!.attempts, 2); + assert.equal(flaky!.seconds, 10); + assert.equal(flaky!.error, 'tap failed; the runner session ended'); + assert.equal(flaky!.failurePage, join(dir, 'failures', 'specs_golden_auth-start_opens-opens-eeeeeeee.md')); + assert.equal(flaky!.failureScreen, join(dir, 'artifacts', 'ios/o/attempt-0/screen.txt')); + assert.equal(flaky!.failureScreenshot, join(dir, 'artifacts', 'ios/o/attempt-0/screenshots/001-failure.png')); + + const [failed] = parseE2EReport(retried('failed', { status: 'failed', error: { message: 'tap failed again' }, ...attempt('ios/o/attempt-1') }), opens, dir); + assert.equal(failed!.status, 'failed'); + assert.equal(failed!.attempts, 2); + assert.equal(failed!.error, 'tap failed again', 'a test that fails every attempt shows its last one'); + assert.equal(failed!.failureScreenshot, join(dir, 'artifacts', 'ios/o/attempt-1/screenshots/001-failure.png')); + }); + + it('maps statuses, skip reasons, errors, and failure pages, in the directory its invocation wrote to', () => { + const run = mkdtempSync(join(tmpdir(), 'verify-report-')) as EvidencePath; + const dir = e2eOutputDir(run, 1); + assert.equal(dir, join(run, 'e2e-2')); + mkdirSync(join(dir, 'failures'), { recursive: true }); + writeFileSync(join(dir, 'failures', 'specs_explored_probe-fails-cccccccc.md'), ''); + mkdirSync(join(dir, 'artifacts', 'ios/p/attempt-0'), { recursive: true }); + writeFileSync(join(dir, 'artifacts', 'ios/p/attempt-0/screen.txt'), ''); + const results = parseE2EReport(report, ['specs/golden/auth-start/opens.e2e.ts', 'specs/golden/sign-up/complete.e2e.ts', 'specs/golden/sign-up/request-code.e2e.ts', 'specs/explored/probe.e2e.ts'].map((path) => ({ kind: 'golden', path, feature: null })), dir); + assert.deepEqual(results.map((r) => r.status), ['passed', 'skipped', 'skipped', 'failed', 'failed']); + assert.equal(results[3]!.error, 'not run: infrastructure-unavailable the device could not be opened'); + assert.equal(results[0]!.seconds, 9.1); + assert.equal(results[0]!.spec.feature, null); + assert.equal(results[1]!.skipReason, 'filtered: title does not match --grep'); + assert.equal(results[1]!.skippedBy, null); + assert.equal(results[2]!.skipReason, 'skipped: ios only', 'a platform-scoped spec is skipped, not failed'); + assert.equal(results[2]!.skippedBy, 'platform'); + assert.equal(results[4]!.error, 'expect.toBeVisible failed; observed: no node'); + assert.equal(results[4]!.failureScreen, join(dir, 'artifacts', 'ios/p/attempt-0/screen.txt')); + assert.equal(results[4]!.failureScreenshot, join(dir, 'artifacts', 'ios/p/attempt-0/screenshots/001-failure.png'), 'the full path, never truncated'); + const selected = parseE2EReport(report, [{ kind: 'golden', path: 'specs/golden/auth-start/opens.e2e.ts', feature: null }], dir); + assert.deepEqual(selected.map((r) => r.spec.path), ['specs/golden/auth-start/opens.e2e.ts', 'specs/golden/auth-start/opens.e2e.ts'], 'files e2e lists but the run did not select are dropped'); + assert.equal(results[4]!.failurePage, join(dir, 'failures', 'specs_explored_probe-fails-cccccccc.md')); + }); + + it('refuses a report of another schema', () => { + assert.throws(() => parseE2EReport({ schemaVersion: 'report-2', run: { results: [] } }, [], '/x' as EvidencePath), { code: 'E2E_CRASHED' }); + }); + + it('copies app.screenshot artifacts under their labels, a later group replacing an earlier one with the same label', () => { + const dir = mkdtempSync(join(tmpdir(), 'verify-shots-')) as EvidencePath; + for (const [index, bytes] of [[0, 'first'], [1, 'second']] as const) { + mkdirSync(join(e2eOutputDir(dir, index), 'artifacts', 'ios/x/attempt-0/screenshots'), { recursive: true }); + writeFileSync(join(e2eOutputDir(dir, index), 'artifacts', 'ios/x/attempt-0/screenshots/001-auth.png'), bytes); + assert.deepEqual(collectScreenshots(report, dir, e2eOutputDir(dir, index)), [{ label: 'auth', path: join(dir, 'screenshots', 'auth.png') }]); + assert.equal(readFileSync(join(dir, 'screenshots', 'auth.png'), 'utf8'), bytes); + } + }); +}); + +describe('MANIFEST', () => { + it('matches the files, so doctor reports no drift', () => { + assert.deepEqual(manifestDrift(), []); + }); +}); + diff --git a/integration/expo-native/test/end-run.test.ts b/integration/expo-native/test/end-run.test.ts new file mode 100644 index 00000000000..533976e0744 --- /dev/null +++ b/integration/expo-native/test/end-run.test.ts @@ -0,0 +1,33 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { existsSync, mkdtempSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { VerifyFailure, type EvidencePath, type Recording } from '../src/core/types.ts'; +import { endRun } from '../src/core/verbs.ts'; +import { newEntryId, openWorkspace } from '../src/core/workspace.ts'; + +describe('endRun', () => { + it('stops the broker, removes scratch, and closes the recorder entry even when the recorder fails, then reports the failure', async () => { + const dir = mkdtempSync(join(tmpdir(), 'verify-end-')); + const workspace = openWorkspace({ packageDir: dir, worktree: dir, home: join(dir, 'home') }); + const { scratch } = workspace.newRun(); + const recorderEntry = newEntryId(); + workspace.append({ id: recorderEntry, kind: 'process', what: 'recorder', pid: 1, startedAt: new Date().toISOString() }); + let brokerStopped = false; + const recording: Recording = { + process: { pid: 1, startedAt: 0 }, + stop: async (): Promise => { + throw new VerifyFailure('NOT_READY', 'adb pull failed', 'rerun with --no-video'); + }, + }; + await assert.rejects( + endRun(workspace, { recording, recorderEntry, broker: { stop: async () => void (brokerStopped = true) }, scratch }), + { code: 'NOT_READY', message: 'adb pull failed' }, + ); + assert.equal(brokerStopped, true, 'broker stopped'); + assert.equal(existsSync(scratch), false, 'scratch, with the broker token, removed'); + assert.deepEqual(workspace.unclosedEntries(), [], 'recorder entry closed'); + }); +}); diff --git a/integration/expo-native/test/evidence.test.ts b/integration/expo-native/test/evidence.test.ts new file mode 100644 index 00000000000..dc518e413f7 --- /dev/null +++ b/integration/expo-native/test/evidence.test.ts @@ -0,0 +1,200 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { existsSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { heldInstances } from '../testing/fake-instances.ts'; +import { describe, it } from 'node:test'; +import { newTestEmail } from '../specs/support/clerk.ts'; +import { assertPublishable, loggedUserIds, sealEvidence } from '../src/core/evidence.ts'; +import type { Runner } from '../src/core/exec.ts'; +import { commentBody, postToPullRequest } from '../src/core/publish.ts'; +import { Secret } from '../specs/support/secret.ts'; +import { attach } from '../src/core/verbs.ts'; +import { newRunId } from '../specs/support/inputs.ts'; +import { openWorkspace } from '../src/core/workspace.ts'; +import type { BuildKey, EvidencePath, EvidenceRecord, HostAdapter, RunId } from '../src/core/types.ts'; + +const OWN_USER = 'user_own'; + +const hostLogLine = (userId: string | null): string => + `2026-10-03 00:00:01.000 Df E2EHost[1:1] [com.clerk.verify:state] verify {"launchId":"l1","screen":"home","signedIn":${userId !== null},"ticket":"succeeded"${userId === null ? '' : `,"userId":"${userId}"`},"v":1}\n`; + +function partialRecord(dir: EvidencePath, run: RunId): Omit { + return { + run, + startedAt: '2026-10-03T00:00:00.000Z', + finishedAt: '2026-10-03T00:00:10.000Z', + repo: 'clerk-ios', + gitHead: 'abc', + dirty: false, + platform: 'ios', + backend: 'local', + device: 'verify-ios-1', + build: 'ios-000000000000' as BuildKey, + results: [ + { spec: { kind: 'explored', path: 'specs/explored/a.e2e.ts', feature: null }, title: 'a', platform: 'ios', status: 'passed', seconds: 1, attempts: 1, error: null, skipReason: null, skippedBy: null, tags: [], failurePage: null, failureScreen: null, failureScreenshot: null }, + ], + videos: [join(dir, 'video.mp4') as EvidencePath], + screenshots: [{ label: 'profile', path: join(dir, 'screenshots', 'profile.png') as EvidencePath }], + appLog: join(dir, 'app.log') as EvidencePath, + e2eReport: join(dir, 'e2e', 'report.json') as EvidencePath, + identities: [{ email: newTestEmail(run), userId: OWN_USER }], + settings: [{ label: 'standard', askedBy: null, specs: ['specs/explored/a.e2e.ts'], application: null, changed: false, held: true, e2eReport: join(dir, 'e2e', 'report.json') as EvidencePath }], + }; +} + +function runDir(): { dir: EvidencePath; run: RunId } { + const run = newRunId(); + const dir = join(mkdtempSync(join(tmpdir(), 'verify-evidence-')), run) as EvidencePath; + mkdirSync(join(dir, 'e2e'), { recursive: true }); + mkdirSync(join(dir, 'screenshots'), { recursive: true }); + writeFileSync(join(dir, 'video.mp4'), 'mp4'); + writeFileSync(join(dir, 'screenshots', 'profile.png'), 'png'); + writeFileSync(join(dir, 'app.log'), `log line\n${hostLogLine(null)}${hostLogLine(OWN_USER)}${hostLogLine(OWN_USER)}`); + writeFileSync(join(dir, 'e2e', 'report.json'), '{}'); + return { dir, run }; +} + +describe('sealEvidence', () => { + it('writes a sealed run.json beside the evidence layout', () => { + const { dir, run } = runDir(); + const record = sealEvidence(dir, partialRecord(dir, run), []); + for (const file of ['run.json', 'video.mp4', 'screenshots/profile.png', 'app.log', 'e2e/report.json']) { + assert.ok(existsSync(join(dir, file)), file); + } + const written = JSON.parse(readFileSync(join(dir, 'run.json'), 'utf8')) as EvidenceRecord; + assert.equal(written.sealed, true); + assert.deepEqual(written.tainted, []); + assert.deepEqual(record, written); + }); + + it('marks a file holding a used secret as tainted without rewriting it', () => { + const { dir, run } = runDir(); + const ticket = new Secret('ticket', 'tkt_planted_secret_value'); + const plain = ticket.use('launch-argument', (value) => value); + writeFileSync(join(dir, 'e2e', 'report.json'), `{"label":"-verifySignInTicket ${plain}"}`); + const record = sealEvidence(dir, partialRecord(dir, run)); + assert.deepEqual(record.tainted, [join(dir, 'e2e', 'report.json')]); + assert.ok(readFileSync(join(dir, 'e2e', 'report.json'), 'utf8').includes(plain), 'sealing leaves e2e files as written'); + assert.throws(() => assertPublishable(record, [OWN_USER]), { code: 'EVIDENCE_UNSAFE', message: /secret/ }); + }); +}); + +describe('assertPublishable', () => { + it('passes a clean run whose app log names only its own users', () => { + const { dir, run } = runDir(); + const record = sealEvidence(dir, partialRecord(dir, run), []); + assert.deepEqual(loggedUserIds(dir), [OWN_USER]); + assert.equal(assertPublishable(record, loggedUserIds(dir)).run, run); + }); + + it('rejects a run whose app log names a user the run did not create', () => { + const { dir, run } = runDir(); + const record = sealEvidence(dir, partialRecord(dir, run), []); + writeFileSync(join(dir, 'app.log'), `${hostLogLine(OWN_USER)}${hostLogLine('user_someone_else')}`); + assert.throws(() => assertPublishable(record, loggedUserIds(dir)), { code: 'EVIDENCE_UNSAFE', message: /user_someone_else/ }); + }); + + it('reads a user ID from a host line that has spaces around the colon', () => { + const { dir } = runDir(); + writeFileSync(join(dir, 'app.log'), 'I ClerkVerify: verify {"signedIn": true, "userId" : "user_spaced", "v": 1}\n'); + assert.deepEqual(loggedUserIds(dir), ['user_spaced']); + }); + + it('reads no user from a run that kept no app log', () => { + const { dir } = runDir(); + rmSync(join(dir, 'app.log')); + assert.deepEqual(loggedUserIds(dir), []); + }); + + it('rejects a run with a failing spec', () => { + const { dir, run } = runDir(); + const base = sealEvidence(dir, partialRecord(dir, run), []); + const failing = { ...base, results: base.results.map((r) => ({ ...r, status: 'failed' as const })) }; + assert.throws(() => assertPublishable(failing, []), { code: 'EVIDENCE_UNSAFE', message: /failing/ }); + }); +}); + +describe('the comment attach posts', () => { + it('counts a test that passed only on a retry apart from the passed ones', () => { + const { dir, run } = runDir(); + const base = partialRecord(dir, run); + const record = sealEvidence(dir, { ...base, results: [base.results[0]!, { ...base.results[0]!, title: 'b', status: 'flaky', attempts: 2, error: 'tap failed' }] }, []); + const body = commentBody(assertPublishable(record, [OWN_USER])); + assert.match(body, /, 1 of 2 passed, 1 flaky \(passed only on a retry\)\.$/m); + assert.match(body, /^- flaky: `specs\/explored\/a\.e2e\.ts` b$/m); + }); +}); + +describe('assertPublishable and the groups of a run', () => { + it('rejects a run with a group that lost its settings or never invoked e2e, though every result passed', () => { + const { dir, run } = runDir(); + const base = sealEvidence(dir, partialRecord(dir, run), []); + const group = base.settings[0]!; + const other = { ...group, label: 'auth_multi_factor.required_for_sign_up=true' }; + assert.throws(() => assertPublishable({ ...base, settings: [group, { ...other, held: false }] }, []), { code: 'EVIDENCE_UNSAFE', message: /1 group that did not run in full on its settings: auth_multi_factor\.required_for_sign_up=true/ }); + assert.throws(() => assertPublishable({ ...base, settings: [{ ...group, e2eReport: null }, other] }, []), { code: 'EVIDENCE_UNSAFE', message: /did not run in full on its settings: standard/ }); + }); +}); + +describe('attach', () => { + const host = { repo: 'clerk-ios', githubRepo: 'clerk/clerk-ios' } as HostAdapter; + + function recordingRunner(attachFlag = true): { runner: Runner; calls: (readonly string[])[] } { + const calls: (readonly string[])[] = []; + const runner: Runner = async (command, args) => { + if (args.includes('--help')) return { code: 0, stdout: attachFlag ? ' --attach file Attach a file\n' : ' -b, --body text The comment body text\n', stderr: '' }; + calls.push([command, ...args]); + return { code: 0, stdout: 'https://github.com/clerk/clerk-ios/pull/9#issuecomment-1\n', stderr: '' }; + }; + return { runner, calls }; + } + + it('never calls gh for a run that fails the gate', async () => { + const packageDir = mkdtempSync(join(tmpdir(), 'verify-attach-')); + const workspace = openWorkspace({ packageDir, worktree: packageDir, home: join(packageDir, 'home') }); + const { run, dir } = workspace.newRun(); + writeFileSync(join(dir, 'video.mp4'), 'x'); + writeFileSync(join(dir, 'app.log'), hostLogLine('user_foreign')); + sealEvidence(dir, partialRecord(dir, run), []); + const { runner, calls } = recordingRunner(); + const deps = { host, workspace, runner, env: {}, progress: () => undefined, instances: heldInstances() }; + await assert.rejects(attach(deps, { verb: 'attach', run, pr: 9, screenshots: 'all' }), { code: 'EVIDENCE_UNSAFE' }); + assert.equal(calls.length, 0); + }); + + it('posts once with --repo and --attach, then reports alreadyPosted', async () => { + const { dir, run } = runDir(); + const record = sealEvidence(dir, partialRecord(dir, run), []); + const publishable = assertPublishable(record, [OWN_USER]); + const { runner, calls } = recordingRunner(); + const first = await postToPullRequest(publishable, dir, host, 9, 'all', runner); + const second = await postToPullRequest(publishable, dir, host, 9, 'all', runner); + assert.equal(calls.length, 1); + const args = calls[0]!; + assert.deepEqual(args.slice(0, 6), ['gh', 'pr', 'comment', '9', '--repo', 'clerk/clerk-ios']); + assert.deepEqual(args.filter((_, i) => args[i - 1] === '--attach'), [join(dir, 'video.mp4'), join(dir, 'screenshots', 'profile.png')]); + assert.equal(first.alreadyPosted, false); + assert.equal(second.alreadyPosted, true); + assert.equal(second.commentUrl, 'https://github.com/clerk/clerk-ios/pull/9#issuecomment-1'); + assert.ok(existsSync(join(dir, 'posted-9.json'))); + const other = await postToPullRequest(publishable, dir, host, 10, 'all', runner); + assert.equal(other.alreadyPosted, false); + assert.equal(calls.length, 2); + assert.equal(calls[1]![3], '10'); + }); + + it('posts nothing and names the fix when gh pr comment has no --attach', async () => { + const { dir, run } = runDir(); + const publishable = assertPublishable(sealEvidence(dir, partialRecord(dir, run), []), [OWN_USER]); + const { runner, calls } = recordingRunner(false); + await assert.rejects(postToPullRequest(publishable, dir, host, 9, 'all', runner), { + code: 'NOT_READY', + message: `this gh has no \`gh pr comment --attach\`, so the video and screenshots of run ${run} cannot be posted`, + fix: 'install a gh build whose `gh pr comment` has --attach', + }); + assert.deepEqual(calls, []); + assert.equal(existsSync(join(dir, 'posted-9.json')), false); + }); +}); diff --git a/integration/expo-native/test/fill.test.ts b/integration/expo-native/test/fill.test.ts new file mode 100644 index 00000000000..6452bfaf9b2 --- /dev/null +++ b/integration/expo-native/test/fill.test.ts @@ -0,0 +1,160 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { describe, it } from 'node:test'; +import { busyWaits } from '../specs/support/busy-runner.ts'; +import { fillField, type FillDevice, type NamedField } from '../specs/support/fill.ts'; +import type { Platform } from '../specs/support/types.ts'; + +const BUSY = 'ENGINE_FAILURE: perform tap failed: the iOS automation runner is still finishing a command that overran its watchdog (session e2e-ios-0 on iPhone 17 Pro)'; + +interface BusyRunner { + readonly tap?: number; + readonly read?: number; + readonly type?: number; + readonly replace?: number; +} + +interface Screen { + texts(): readonly string[]; + afterTyping?(held: string): string; +} + +const CODE = '424242'; +const resendIn = (seconds: number): string => `Didn't receive a code? Resend (${seconds})`; +const EMAIL_CODE_SCREEN = ['Check your email', 'someone+clerk_test@example.com', resendIn(30)]; +const PHONE_CODE_SCREEN = ['Check your phone', '+1 201-555-0160', resendIn(30)]; + +function form(runner: BusyRunner = {}, platform: Platform = 'ios', tapsUntilFocused = 1, screen: Screen = { texts: () => EMAIL_CODE_SCREEN }) { + const refusalsLeft = { tap: runner.tap ?? 0, read: runner.read ?? 0, type: runner.type ?? 0, replace: runner.replace ?? 0 }; + const calls: string[] = []; + let held = ''; + let focused = false; + const step = (site: keyof typeof refusalsLeft, what: string): void => { + calls.push(what); + if (refusalsLeft[site] > 0) { + refusalsLeft[site] -= 1; + throw new Error(BUSY); + } + }; + const frame = { x: 20, y: 300, width: 200, height: 40 }; + const field: NamedField = { + tap: async () => { + step('tap', 'tap'); + focused = (tapsUntilFocused -= 1) <= 0; + }, + all: async () => [{ inputValue: async () => held, boundingBox: async () => frame }], + }; + const device: FillDevice = { + platform, + focused: { + count: async () => (step('read', 'read'), focused ? 1 : 0), + boundingBox: async () => frame, + inputValue: async () => held, + tap: async () => step('tap', 'refocus'), + }, + texts: { allTextContents: async () => [...screen.texts()] }, + type: async ([command, ...operands]) => { + step(command === 'fill' ? 'replace' : 'type', `${command} ${operands.join(' ')}`); + held = command === 'fill' ? operands.at(-1)! : held + operands.join(''); + held = screen.afterTyping?.(held) ?? held; + }, + }; + let waited = 0; + const fill = (text: string, most: number): Promise => + fillField(field, text, device, { tapTimeoutMs: 1_000, reads: 2, now: () => 0, wait: async () => undefined }, busyWaits(most, async () => void (waited += 1))); + return { fill, calls, held: () => held, waited: () => waited }; +} + +describe('one host.fill', () => { + it('taps the field, types the text on the device, and reads back that the field holds it', async () => { + const email = form(); + await email.fill('someone+clerk_test@example.com', 6); + assert.equal(email.held(), 'someone+clerk_test@example.com'); + assert.deepEqual(email.calls.filter((call) => !call.startsWith('read')), ['tap', 'type someone+clerk_te', 'type st@example.com']); + assert.equal(email.waited(), 0); + }); + + it('waits for a busy runner at the tap, at a read, and at the typing, then replaces what the busy typing may have left', async () => { + const code = form({ tap: 2, read: 1, type: 1 }); + await code.fill('424242', 6); + assert.equal(code.held(), '424242'); + assert.equal(code.waited(), 4); + assert.equal(code.calls.at(-1)?.startsWith('read'), true); + assert.ok(code.calls.includes('fill 120 320 424242'), 'the replacement goes to the middle of the focused field'); + }); + + it('spends one bound of waits on all its steps together: steps that need five waits fail under a bound of four and pass under five', async () => { + const needsFive = { tap: 2, type: 1, replace: 2 }; + const short = form(needsFive); + await assert.rejects(short.fill('424242', 4), (error: Error) => error.message === BUSY); + assert.equal(short.waited(), 4); + const enough = form(needsFive); + await enough.fill('424242', 5); + assert.deepEqual([enough.held(), enough.waited()], ['424242', 5]); + }); + + it('fails at the first refusal once the bound is spent, and types nothing after a tap that never landed', async () => { + const stuck = form({ tap: 7 }); + await assert.rejects(stuck.fill('424242', 6), (error: Error) => error.message === BUSY); + assert.deepEqual([stuck.waited(), stuck.held()], [6, '']); + assert.deepEqual(stuck.calls, Array.from({ length: 7 }, () => 'tap')); + }); + + it('taps the field again on Android when the first tap left nothing focused, so no text is typed at nothing', async () => { + const android = form({}, 'android', 2); + await android.fill('424242', 6); + assert.equal(android.held(), '424242'); + assert.deepEqual(android.calls.filter((call) => !call.startsWith('read')), ['tap', 'tap', 'type 424242']); + }); + + describe('when the field reads empty after the typing', () => { + const swallowsTheFirstTyping = (): ((held: string) => string) => { + let typings = 0; + return (held) => ((typings += 1) === 1 ? '' : held); + }; + const typedTwice = ['tap', `type ${CODE}`, 'refocus', `type ${CODE}`]; + + it('types a code once when the app took it and shows the next screen, whose empty code field has the same name in the same place', async () => { + const screens = [EMAIL_CODE_SCREEN, PHONE_CODE_SCREEN, ['Signed in as someone+clerk_test@example.com']]; + let on = 0; + const emailCode = form({}, 'ios', 1, { texts: () => screens[on]!, afterTyping: (held) => (held === CODE ? ((on += 1), '') : held) }); + await emailCode.fill(CODE, 6); + assert.deepEqual(emailCode.calls.filter((call) => !call.startsWith('read')), ['tap', `type ${CODE}`]); + assert.deepEqual(screens[on], PHONE_CODE_SCREEN, 'the phone code screen is still waiting for its code'); + }); + + it('types again when the typing never arrived: the screen shows nothing it did not show before the typing', async () => { + const dropped = form({}, 'ios', 1, { texts: () => EMAIL_CODE_SCREEN, afterTyping: swallowsTheFirstTyping() }); + await dropped.fill(CODE, 6); + assert.deepEqual(dropped.calls.filter((call) => !call.startsWith('read')), typedTwice); + assert.equal(dropped.held(), CODE); + }); + + it('takes a screen whose resend countdown went on for the same screen, so a typing that never arrived there is typed again', async () => { + let seconds = 30; + const counting = form({}, 'ios', 1, { texts: () => [...EMAIL_CODE_SCREEN.slice(0, 2), resendIn((seconds -= 1))], afterTyping: swallowsTheFirstTyping() }); + await counting.fill(CODE, 6); + assert.deepEqual(counting.calls.filter((call) => !call.startsWith('read')), typedTwice); + assert.equal(counting.held(), CODE); + }); + + it('types again when texts only left the screen since the typing, as the texts of the screen before do while an iOS screen arrives', async () => { + let typings = 0; + const arriving = form({}, 'ios', 1, { texts: () => (typings === 0 ? ['Welcome! Sign in to continue', ...EMAIL_CODE_SCREEN] : EMAIL_CODE_SCREEN), afterTyping: (held) => ((typings += 1) === 1 ? '' : held) }); + await arriving.fill(CODE, 6); + assert.deepEqual(arriving.calls.filter((call) => !call.startsWith('read')), typedTwice); + assert.equal(arriving.held(), CODE); + }); + + it('types once and claims nothing when the texts of the screen cannot be read', async () => { + const unreadable = form({}, 'ios', 1, { + texts: () => { + throw new Error('ENGINE_FAILURE: the snapshot holds a secure node'); + }, + afterTyping: swallowsTheFirstTyping(), + }); + await unreadable.fill(CODE, 6); + assert.deepEqual(unreadable.calls.filter((call) => !call.startsWith('read')), ['tap', `type ${CODE}`]); + }); + }); +}); diff --git a/integration/expo-native/test/freshness.test.ts b/integration/expo-native/test/freshness.test.ts new file mode 100644 index 00000000000..410c9b79843 --- /dev/null +++ b/integration/expo-native/test/freshness.test.ts @@ -0,0 +1,451 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { mkdirSync, mkdtempSync, symlinkSync, utimesSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { + changedFiles, + confirmServed, + fingerprint, + isBundledOutput, + isPackageSource, + isStale, + isTsdownSource, + listFiles, + newest, + staleInScope, + staleOutOfScope, + workspaceDependencies, + judge, + type BundleView, + type Fetched, + type Fingerprint, + type GateIO, + type GateMemory, +} from '../src/freshness.ts'; + +const scratch = () => mkdtempSync(join(tmpdir(), 'verify-fresh-')); + +describe('which sources the builds read', () => { + it('counts only the files tsdown builds for @clerk/expo', () => { + assert.equal(isTsdownSource('hooks/useAuth.ts'), true); + assert.equal(isTsdownSource('native/AuthView.tsx'), true); + assert.equal(isTsdownSource('hooks/useAuth.test.ts'), false); + assert.equal(isTsdownSource('hooks/__tests__/useAuth.ts'), false); + assert.equal(isTsdownSource('hooks/.useAuth.ts.swp'), false); + }); + + it('ignores tests, specs, and declarations in a sibling package', () => { + assert.equal(isPackageSource('index.ts'), true); + assert.equal(isPackageSource('index.spec.ts'), false); + assert.equal(isPackageSource('types.d.ts'), false); + assert.equal(isPackageSource('__tests__/x.ts'), false); + }); + + it('ignores an edited test file when deciding whether dist caught up', () => { + const root = scratch(); + mkdirSync(join(root, 'src', 'hooks'), { recursive: true }); + mkdirSync(join(root, 'dist', 'hooks'), { recursive: true }); + writeFileSync(join(root, 'src', 'hooks', 'useAuth.ts'), ''); + writeFileSync(join(root, 'dist', 'hooks', 'useAuth.js'), ''); + writeFileSync(join(root, 'src', 'hooks', 'useAuth.test.ts'), ''); + utimesSync(join(root, 'src', 'hooks', 'useAuth.ts'), 100, 100); + utimesSync(join(root, 'dist', 'hooks', 'useAuth.js'), 200, 200); + utimesSync(join(root, 'src', 'hooks', 'useAuth.test.ts'), 300, 300); + const src = newest(listFiles(join(root, 'src'), '', isTsdownSource)); + const dist = newest(listFiles(join(root, 'dist'), '', rel => rel.endsWith('.js'))); + assert.ok(dist >= src); + }); + + it('treats .cjs and .mjs chunks as bundled output, and not maps or declarations', () => { + for (const rel of ['index.js', 'internal.cjs', 'hooks-ByQmbuum.cjs', 'index.mjs']) + assert.equal(isBundledOutput(rel), true, rel); + for (const rel of ['index.js.map', 'index.d.ts', 'errors.d.cts']) assert.equal(isBundledOutput(rel), false, rel); + }); + + it('flags a sibling whose source is newer than its dist', () => { + assert.equal(isStale([{ rel: 'a.ts', mtime: 300 }], [{ rel: 'a.js', mtime: 200 }]), true); + assert.equal(isStale([{ rel: 'a.ts', mtime: 100 }], [{ rel: 'a.js', mtime: 200 }]), false); + assert.equal(isStale([{ rel: 'a.ts', mtime: 300 }], []), false); + }); +}); + +describe('workspaceDependencies', () => { + it('follows runtime dependencies into the workspace, transitively, and skips dev dependencies', () => { + const root = scratch(); + const pkg = (dir: string, json: object) => { + mkdirSync(join(root, 'packages', dir, 'node_modules', '@clerk'), { recursive: true }); + writeFileSync(join(root, 'packages', dir, 'package.json'), JSON.stringify(json)); + }; + pkg('expo', { + name: '@clerk/expo', + dependencies: { '@clerk/react': '*' }, + peerDependencies: { '@clerk/expo-passkeys': '*' }, + devDependencies: { '@clerk/testing': '*' }, + }); + pkg('react', { name: '@clerk/react', dependencies: { '@clerk/shared': '*' } }); + pkg('shared', { name: '@clerk/shared' }); + pkg('expo-passkeys', { name: '@clerk/expo-passkeys' }); + pkg('testing', { name: '@clerk/testing' }); + const link = (from: string, name: string, to: string) => + symlinkSync(join(root, 'packages', to), join(root, 'packages', from, 'node_modules', '@clerk', name)); + link('expo', 'react', 'react'); + link('expo', 'expo-passkeys', 'expo-passkeys'); + link('expo', 'testing', 'testing'); + link('react', 'shared', 'shared'); + const names = workspaceDependencies(root, join(root, 'packages', 'expo')) + .map(p => p.name) + .sort(); + assert.deepEqual(names, ['@clerk/expo-passkeys', '@clerk/react', '@clerk/shared']); + }); +}); + +describe('judge', () => { + const rel = 'packages/expo/dist/hooks/useAuth.js'; + const body = `var x;__d(function(){},12,[],"../../../${rel}");`; + const fp = (hash: string, since: number): Fingerprint => ({ [rel]: { mtime: since, size: 1, hash, since } }); + const fresh: GateMemory = { metroPid: 1, spawn: null, seen: {}, outputs: {} }; + + it('trusts the first revision of a Metro it started when dist has not changed since', () => { + const memory = { ...fresh, spawn: fp('a', 1_000) }; + assert.equal(judge(memory, fp('a', 1_000), { revId: 'r1', lastModified: 2_000, body }).verdict, 'fresh'); + }); + + it('restarts a Metro whose first bundle cannot be dated against the current dist', () => { + assert.equal( + judge({ ...fresh, spawn: fp('a', 1_000) }, fp('b', 3_000), { revId: 'r1', lastModified: 4_000, body }).verdict, + 'restart', + ); + assert.equal(judge(fresh, fp('a', 1_000), { revId: 'r1', lastModified: 4_000, body }).verdict, 'restart'); + }); + + it('does not confirm a revision it already saw paired with older content', () => { + const afterEdit1 = judge({ ...fresh, spawn: fp('a', 1_000) }, fp('a', 1_000), { + revId: 'r1', + lastModified: 2_000, + body, + }); + assert.equal(afterEdit1.verdict, 'fresh'); + const lagging = judge(afterEdit1.memory, fp('b', 5_400), { revId: 'r1', lastModified: 2_000, body }); + assert.equal(lagging.verdict, 'stale'); + assert.equal(judge(lagging.memory, fp('b', 5_400), { revId: 'r2', lastModified: 6_000, body }).verdict, 'fresh'); + }); + + it('rejects a revision first seen now but dated before the current content', () => { + const memory = { ...fresh, seen: { r1: 'x' } }; + const early = judge(memory, fp('c', 9_400), { revId: 'r2', lastModified: 8_000, body }); + assert.equal(early.verdict, 'stale'); + assert.equal(judge(early.memory, fp('c', 9_400), { revId: 'r2', lastModified: 8_000, body }).verdict, 'stale'); + }); + + it('keeps confirming the same revision after a rewrite with identical content', () => { + const first = judge({ ...fresh, spawn: fp('a', 1_000) }, fp('a', 1_000), { + revId: 'r1', + lastModified: 2_000, + body, + }); + const rewritten = { [rel]: { mtime: 7_000, size: 1, hash: 'a', since: 1_000 } }; + assert.equal(judge(first.memory, rewritten, { revId: 'r1', lastModified: 2_000, body }).verdict, 'fresh'); + }); + + it('ignores a changed file the bundle does not include', () => { + const web = 'packages/expo/dist/web/index.js'; + const first = judge({ ...fresh, spawn: fp('a', 1_000) }, fp('a', 1_000), { + revId: 'r1', + lastModified: 2_000, + body, + }); + const withWeb = { ...fp('a', 1_000), [web]: { mtime: 8_000, size: 1, hash: 'w', since: 8_000 } }; + assert.equal(judge(first.memory, withWeb, { revId: 'r1', lastModified: 2_000, body }).verdict, 'fresh'); + }); +}); + +describe('fingerprint', () => { + it('keeps the time content first appeared across an identical rewrite', () => { + const root = scratch(); + const rel = 'a.js'; + writeFileSync(join(root, rel), 'one'); + utimesSync(join(root, rel), 10, 10); + const first = fingerprint(root, [rel], null); + writeFileSync(join(root, rel), 'one'); + utimesSync(join(root, rel), 20, 20); + const second = fingerprint(root, [rel], first); + assert.equal(second[rel]!.since, first[rel]!.since); + assert.deepEqual(changedFiles(first, second), []); + writeFileSync(join(root, rel), 'two'); + utimesSync(join(root, rel), 30, 30); + assert.equal(fingerprint(root, [rel], second)[rel]!.since, 30_000); + }); +}); + +describe('confirmServed', () => { + const rel = 'packages/expo/dist/hooks/useAuth.js'; + const body = `__d(function(){},1,[],"../../../${rel}");`; + type Step = { readonly files: Record; readonly response: Fetched }; + const harness = (steps: Step[], after: Record[] = []) => { + let clock = 0; + let call = 0; + let files: Record = steps[0]!.files; + const touched: string[][] = []; + let restarts = 0; + const io: GateIO = { + list: () => Object.keys(files), + fingerprint: (rels, previous) => + Object.fromEntries( + rels + .filter(r => files[r] !== undefined) + .map(r => { + const hash = files[r]!; + const before = previous?.[r]; + return [ + r, + { + mtime: clock, + size: hash.length, + hash, + since: before !== undefined && before.hash === hash ? before.since : clock, + }, + ]; + }), + ), + fetch: async () => { + const step = steps[Math.min(call, steps.length - 1)]!; + files = after[call] ?? step.files; + call += 1; + return step.response; + }, + touch: rels => touched.push([...rels]), + restart: async () => { + restarts += 1; + return 2; + }, + now: () => clock, + sleep: async ms => { + clock += ms; + }, + progress: () => undefined, + }; + return { io, touched: () => touched, restarts: () => restarts, calls: () => call }; + }; + const ok = (revId: string, lastModified: number): Fetched => ({ + ok: true, + value: { revId, lastModified, body }, + }); + const options = { timeoutMs: 20_000, nudgeAfterMs: 5_000, maxRestarts: 2, settledFailureMs: 3_000 }; + + it('confirms after two matching reads of a fresh revision', async () => { + const h = harness([{ files: { [rel]: 'a' }, response: ok('r1', 0) }]); + const result = await confirmServed( + h.io, + { metroPid: 1, spawn: { [rel]: { mtime: 0, size: 1, hash: 'a', since: 0 } }, seen: {}, outputs: {} }, + options, + ); + assert.equal(result.ok, true); + assert.equal(h.calls(), 2); + }); + + it('waits out a 500 while tsdown empties and refills dist, then confirms', async () => { + const error: Fetched = { ok: false, transient: false, message: 'bundle 500: UnableToResolveError' }; + const h = harness( + [ + { files: { [rel]: 'a' }, response: error }, + { files: {}, response: error }, + { files: { [rel]: 'b' }, response: ok('r2', 60_000) }, + ], + [{}, { [rel]: 'b' }], + ); + const memory: GateMemory = { + metroPid: 1, + spawn: null, + seen: { r1: 'x' }, + outputs: { [rel]: { mtime: 0, size: 1, hash: 'a', since: 0 } }, + }; + const result = await confirmServed(h.io, memory, options); + assert.equal(result.ok, true, JSON.stringify(result)); + }); + + it('fails a 500 only once it has persisted over settled outputs for the settle window', async () => { + const error: Fetched = { ok: false, transient: false, message: 'bundle 500: SyntaxError' }; + const h = harness([{ files: { [rel]: 'a' }, response: error }]); + const result = await confirmServed(h.io, { metroPid: 1, spawn: null, seen: { r0: 'x' }, outputs: {} }, options); + assert.deepEqual(result, { ok: false, kind: 'bundle-error', message: 'bundle 500: SyntaxError' }); + assert.ok(h.io.now() >= 3_000, `failed after ${h.io.now()}ms`); + assert.ok(h.calls() >= 3); + }); + + it('confirms when a 500 clears within the settle window', async () => { + const error: Fetched = { ok: false, transient: false, message: 'bundle 500: UnableToResolveError' }; + const h = harness([ + { files: { [rel]: 'a' }, response: error }, + { files: { [rel]: 'a' }, response: error }, + { files: { [rel]: 'a' }, response: ok('r1', 0) }, + ]); + const memory: GateMemory = { + metroPid: 1, + spawn: { [rel]: { mtime: 0, size: 1, hash: 'a', since: 0 } }, + seen: {}, + outputs: {}, + }; + assert.equal((await confirmServed(h.io, memory, options)).ok, true); + }); + + it('does not confirm a lagging revision after a second edit, and touches the changed file', async () => { + const memory: GateMemory = { + metroPid: 1, + spawn: null, + seen: { r1: 'digest-of-edit-1' }, + outputs: { [rel]: { mtime: 0, size: 1, hash: 'edit-1', since: 0 } }, + }; + const h = harness([{ files: { [rel]: 'edit-2' }, response: ok('r1', 0) }]); + const result = await confirmServed(h.io, memory, options); + assert.equal(result.ok, false); + assert.ok(h.touched().length > 0); + assert.deepEqual(h.touched()[0], [rel]); + }); + + it('restarts a Metro with no history, at most maxRestarts times', async () => { + const h = harness([{ files: { [rel]: 'a' }, response: ok('r1', 0) }]); + const result = await confirmServed(h.io, { metroPid: 1, spawn: null, seen: {}, outputs: {} }, options); + assert.equal(h.restarts(), 1); + assert.equal(result.ok, true); + }); +}); + +describe('scope', () => { + const workspace = () => { + const root = scratch(); + const pkg = (dir: string, json: object, files: Record, expoModule = false) => { + const base = join(root, 'packages', dir); + mkdirSync(join(base, 'node_modules', '@clerk'), { recursive: true }); + writeFileSync(join(base, 'package.json'), JSON.stringify(json)); + if (expoModule) writeFileSync(join(base, 'expo-module.config.json'), '{}'); + for (const [rel, time] of Object.entries(files)) { + mkdirSync(join(base, rel, '..'), { recursive: true }); + writeFileSync(join(base, rel), rel); + utimesSync(join(base, rel), time, time); + } + }; + pkg( + 'expo', + { + name: '@clerk/expo', + dependencies: { '@clerk/clerk-js': '*', '@clerk/shared': '*', '@clerk/expo-passkeys': '*' }, + }, + { 'src/index.ts': 100, 'dist/index.js': 200 }, + ); + pkg( + 'clerk-js', + { name: '@clerk/clerk-js', dependencies: { '@clerk/shared': '*' } }, + { 'src/index.ts': 100, 'dist/clerk.js': 200 }, + ); + pkg('shared', { name: '@clerk/shared' }, { 'src/index.ts': 100, 'dist/index.cjs': 200 }); + pkg( + 'expo-passkeys', + { name: '@clerk/expo-passkeys', dependencies: { '@clerk/shared': '*' } }, + { 'src/index.ts': 100, 'dist/index.js': 200 }, + true, + ); + const link = (from: string, name: string) => + symlinkSync(join(root, 'packages', name), join(root, 'packages', from, 'node_modules', '@clerk', name)); + link('expo', 'clerk-js'); + link('expo', 'shared'); + link('expo', 'expo-passkeys'); + link('clerk-js', 'shared'); + link('expo-passkeys', 'shared'); + return { + root, + expo: join(root, 'packages', 'expo'), + touch: (rel: string, time: number) => utimesSync(join(root, 'packages', rel), time, time), + }; + }; + + it('finds nothing stale on a fresh build', () => { + const w = workspace(); + assert.deepEqual(staleOutOfScope(w.root, w.expo).stale, []); + assert.deepEqual(staleInScope(w.root, w.expo).stale, []); + }); + + it('refuses an out-of-scope dependency edit, including packages that bundle it', () => { + const w = workspace(); + w.touch('shared/src/index.ts', 300); + assert.deepEqual( + staleOutOfScope(w.root, w.expo) + .stale.map(p => p.name) + .sort(), + ['@clerk/clerk-js', '@clerk/shared'], + ); + assert.deepEqual(staleInScope(w.root, w.expo).stale, []); + }); + + it('still refuses @clerk/clerk-js when only @clerk/shared was rebuilt', () => { + const w = workspace(); + w.touch('shared/src/index.ts', 300); + w.touch('shared/dist/index.cjs', 400); + assert.deepEqual( + staleOutOfScope(w.root, w.expo).stale.map(p => p.name), + ['@clerk/clerk-js'], + ); + }); + + it('clears a content-neutral touch once it has seen the package built', () => { + const w = workspace(); + const { records } = staleOutOfScope(w.root, w.expo); + w.touch('shared/src/index.ts', 300); + const after = staleOutOfScope(w.root, w.expo, records); + assert.deepEqual(after.stale, []); + assert.deepEqual(after.records, records); + }); + + it('still refuses a real edit that a record has seen built from other content', () => { + const w = workspace(); + const { records } = staleOutOfScope(w.root, w.expo); + writeFileSync(join(w.root, 'packages', 'shared', 'src', 'index.ts'), 'changed'); + w.touch('shared/src/index.ts', 300); + assert.deepEqual( + staleOutOfScope(w.root, w.expo, records) + .stale.map(p => p.name) + .sort(), + ['@clerk/clerk-js', '@clerk/shared'], + ); + }); + + it('accepts a revert of an unbuilt edit, because dist was built from the restored content', () => { + const w = workspace(); + const { records } = staleOutOfScope(w.root, w.expo); + const file = join(w.root, 'packages', 'shared', 'src', 'index.ts'); + writeFileSync(file, 'edited'); + w.touch('shared/src/index.ts', 300); + const refused = staleOutOfScope(w.root, w.expo, records); + assert.ok(refused.stale.length > 0); + writeFileSync(file, 'src/index.ts'); + w.touch('shared/src/index.ts', 310); + assert.deepEqual(staleOutOfScope(w.root, w.expo, refused.records).stale, []); + }); + + it('refuses a touch it has no record for, until the build rewrites dist', () => { + const w = workspace(); + w.touch('shared/src/index.ts', 300); + assert.ok(staleOutOfScope(w.root, w.expo).stale.length > 0); + w.touch('shared/dist/index.cjs', 400); + w.touch('clerk-js/dist/clerk.js', 400); + assert.deepEqual(staleOutOfScope(w.root, w.expo).stale, []); + }); + + it('does not rebuild an in-scope sibling again after a content-neutral touch', () => { + const w = workspace(); + const { records } = staleInScope(w.root, w.expo); + w.touch('expo-passkeys/src/index.ts', 300); + assert.deepEqual(staleInScope(w.root, w.expo, records).stale, []); + }); + + it('rebuilds an in-scope Expo module sibling instead of refusing it', () => { + const w = workspace(); + w.touch('expo-passkeys/src/index.ts', 300); + assert.deepEqual( + staleInScope(w.root, w.expo).stale.map(p => p.name), + ['@clerk/expo-passkeys'], + ); + assert.deepEqual(staleOutOfScope(w.root, w.expo).stale, []); + }); +}); diff --git a/integration/expo-native/test/git-env.test.ts b/integration/expo-native/test/git-env.test.ts new file mode 100644 index 00000000000..64ca2068415 --- /dev/null +++ b/integration/expo-native/test/git-env.test.ts @@ -0,0 +1,63 @@ +import { REPOSITORY_LOCAL_GIT_ENV } from '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { execFileSync } from 'node:child_process'; +import { mkdtempSync, readFileSync, readdirSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { describe, it } from 'node:test'; +import { fileURLToPath } from 'node:url'; + +const TEST_DIR = dirname(fileURLToPath(import.meta.url)); +const SCRUB_MODULE = join(TEST_DIR, '..', 'testing', 'git-env.ts'); +const SCRUB_IMPORT = "'../testing/git-env.ts';"; +const AUTHOR = ['-c', 'user.name=t', '-c', 'user.email=t@example.com']; + +function repositoryWithOneCommit(): string { + const dir = mkdtempSync(join(tmpdir(), 'verify-git-env-')); + execFileSync('git', ['init', '-q'], { cwd: dir }); + execFileSync('git', [...AUTHOR, 'commit', '-q', '--allow-empty', '-m', 'first'], { cwd: dir }); + return dir; +} + +function stateOf(repository: string): { readonly head: string; readonly config: string } { + return { + head: execFileSync('git', ['--git-dir', join(repository, '.git'), 'rev-parse', 'HEAD'], { encoding: 'utf8' }).trim(), + config: readFileSync(join(repository, '.git', 'config'), 'utf8'), + }; +} + +describe('git environment of the unit tests', () => { + it('names every variable git treats as local to one repository', () => { + const fromGit = execFileSync('git', ['rev-parse', '--local-env-vars'], { encoding: 'utf8' }).trim().split('\n'); + assert.deepEqual([...REPOSITORY_LOCAL_GIT_ENV].sort(), fromGit.sort()); + }); + + it('is the first import of every test file, so no test runs git against an inherited repository', () => { + const missing = readdirSync(TEST_DIR) + .filter((name) => name.endsWith('.test.ts')) + .filter((name) => !(readFileSync(join(TEST_DIR, name), 'utf8').split('\n')[0] ?? '').endsWith(SCRUB_IMPORT)); + assert.deepEqual(missing, []); + }); + + it('keeps a test that creates and commits to its own repository out of the one GIT_DIR names', () => { + const outer = repositoryWithOneCommit(); + const own = mkdtempSync(join(tmpdir(), 'verify-git-env-own-')); + try { + const before = stateOf(outer); + const script = [ + `await import(${JSON.stringify(SCRUB_MODULE)});`, + `const { execFileSync } = await import('node:child_process');`, + `execFileSync('git', ['init', '-q'], { cwd: ${JSON.stringify(own)} });`, + `execFileSync('git', ${JSON.stringify([...AUTHOR, 'commit', '-q', '--allow-empty', '-m', 'inner'])}, { cwd: ${JSON.stringify(own)} });`, + ].join('\n'); + execFileSync(process.execPath, ['--input-type=module', '-e', script], { + env: { ...process.env, GIT_DIR: join(outer, '.git'), GIT_WORK_TREE: outer, GIT_INDEX_FILE: join(outer, '.git', 'index') }, + }); + assert.deepEqual(stateOf(outer), before); + assert.equal(execFileSync('git', ['log', '--format=%s'], { cwd: own, encoding: 'utf8' }).trim(), 'inner'); + } finally { + rmSync(outer, { recursive: true, force: true }); + rmSync(own, { recursive: true, force: true }); + } + }); +}); diff --git a/integration/expo-native/test/github-report.test.ts b/integration/expo-native/test/github-report.test.ts new file mode 100644 index 00000000000..1f07bbf7eb6 --- /dev/null +++ b/integration/expo-native/test/github-report.test.ts @@ -0,0 +1,249 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { mkdtempSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { createServer, type Server } from 'node:http'; +import type { AddressInfo } from 'node:net'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { after, afterEach, before, beforeEach, describe, it } from 'node:test'; +import type { Report } from 'e2e'; +import { mergeGroupReports, protectGitHubTokens, reportToGitHub } from '../src/core/github-report.ts'; +import { redact } from '../specs/support/secret.ts'; +import { VerifyFailure, type EvidencePath, type EvidenceRecord } from '../src/core/types.ts'; + +interface Row { + readonly file: string; + readonly title: string; + readonly status: 'passed' | 'flaky' | 'failed'; +} + +const ALL: readonly Row[] = [ + { file: 'specs/golden/auth-start/opens.e2e.ts', title: 'opens', status: 'passed' }, + { file: 'specs/golden/sign-up/complete.e2e.ts', title: 'completes', status: 'flaky' }, + { file: 'specs/golden/session-tasks/setup-mfa.e2e.ts', title: 'stops on the task', status: 'passed' }, +]; + +function attempt(index: number, status: 'passed' | 'failed') { + return { + id: `attempt-${index}`, + index, + status, + startedAt: '2026-10-06T13:58:54.750Z', + durationMs: 5000, + artifacts: [], + secondaryErrors: [], + cleanup: 'complete', + steps: [], + ...(status === 'failed' ? { error: { category: 'test', code: 'ASSERTION_FAILED', message: 'the runner session ended', retryable: false } } : {}), + }; +} + +function groupReport(selected: readonly string[], startedAt: string, finishedAt: string): Report { + const results = ALL.map((row, index) => { + const ran = selected.includes(row.file); + return { + id: `${index}`.repeat(64), + testId: `${row.file}::${row.title}`, + kind: 'test', + declarationIndex: 0, + titlePath: [row.title], + file: row.file, + source: { file: row.file, line: 8, column: 1 }, + targetId: 'ios', + platform: 'ios', + agent: 'default', + repeat: 0, + tags: [], + selected: ran, + status: ran ? row.status : 'skipped', + ...(ran ? {} : { skip: { cause: 'filtered', reason: 'file not selected by a positional argument' } }), + attempts: !ran ? [] : row.status === 'flaky' ? [attempt(0, 'failed'), attempt(1, 'passed')] : [attempt(0, row.status)], + }; + }); + return { + schemaVersion: 'report-1', + run: { + id: `run-${startedAt}`, + specVersion: '0.1', + runner: { name: 'e2e', version: '0.18.0' }, + status: 'passed', + exitCode: 0, + startedAt, + finishedAt, + project: { id: 'verify-unit', configDigest: 'digest' }, + environment: { ci: true, trustNoticeShown: false, os: 'darwin', arch: 'arm64', runtime: 'node v24.15.0' }, + targets: [{ id: 'ios', index: 0, platform: 'ios', environment: 'test', engine: { name: 'mobile', version: '0.10.0', spiVersion: 1 }, capabilities: [], artifactCapabilities: [], stateCapability: false }], + serialGroups: [], + results, + errors: [], + summary: { discovered: 3, selected: selected.length, executed: selected.length, passed: selected.length, failed: 0, interrupted: 0, flaky: 0, skipped: 0 }, + limits: { maxAgentContextBytes: 0, maxLedgerBytes: 0, maxObservationBytes: 0, maxEventsPerStep: 0, maxModelTokensPerCall: 0 }, + usage: { discoveredResults: 3, maxAgentContextBytes: 0, maxLedgerBytes: 0, maxObservationBytes: 0, artifactBytes: 0, downloads: 0, events: 0, modelTokens: 0, maxModelCallsInStep: 0, maxActionStepsInStep: 0 }, + }, + } as unknown as Report; +} + +const GROUPS = [ + groupReport([ALL[0]!.file], '2026-10-06T13:58:00.000Z', '2026-10-06T13:58:30.000Z'), + groupReport([ALL[1]!.file], '2026-10-06T13:58:31.000Z', '2026-10-06T13:59:00.000Z'), + groupReport([ALL[2]!.file], '2026-10-06T13:59:01.000Z', '2026-10-06T13:59:20.000Z'), +] as const; + +describe('mergeGroupReports', () => { + it('keeps the result of the group that ran each test, and spans the run from the first group to the last', () => { + const merged = mergeGroupReports(GROUPS, []); + assert.deepEqual(merged.run.results.map((result) => [result.selected, result.status]), [[true, 'passed'], [true, 'flaky'], [true, 'passed']]); + assert.equal(merged.run.startedAt, '2026-10-06T13:58:00.000Z'); + assert.equal(merged.run.finishedAt, '2026-10-06T13:59:20.000Z'); + assert.equal(merged.run.status, 'passed'); + }); + + it('is a failed run when a group did not run, though every report it has passed', () => { + const merged = mergeGroupReports([GROUPS[0]], [{ category: 'infrastructure', code: 'E2E_CRASHED', message: 'MFA required: e2e exited 1', retryable: false }]); + assert.equal(merged.run.status, 'failed'); + assert.equal(merged.run.exitCode, 1); + assert.deepEqual(merged.run.errors.map((error) => error.message), ['MFA required: e2e exited 1']); + }); +}); + +describe('reportToGitHub', () => { + const TOKEN = 'unit-test-github-token-value'; + const VARIABLES = ['GITHUB_ACTIONS', 'GITHUB_REPOSITORY', 'GITHUB_RUN_ID', 'GITHUB_EVENT_NAME', 'GITHUB_EVENT_PATH', 'GITHUB_WORKFLOW', 'GITHUB_JOB', 'GITHUB_SHA', 'GITHUB_REF', 'GITHUB_API_URL', 'GITHUB_STEP_SUMMARY', 'GITHUB_WORKSPACE', 'GITHUB_TOKEN', 'GH_TOKEN']; + const saved = new Map(); + const comments: { id: number; body: string }[] = []; + let requests: string[] = []; + let refuses = false; + let server: Server; + let scratch: string; + + before(async () => { + server = createServer((request, response) => { + let body = ''; + request.on('data', (chunk) => (body += chunk)); + request.on('end', () => { + requests.push(`${request.method} ${request.url?.split('?')[0]} ${request.headers.authorization}`); + const send = (value: unknown) => { + response.writeHead(200, { 'content-type': 'application/json' }); + response.end(JSON.stringify(value)); + }; + if (refuses) { + response.writeHead(403, { 'content-type': 'application/json' }); + return response.end('{"message":"Resource not accessible by integration"}'); + } + if (request.method === 'GET') return send(comments); + if (request.method === 'POST') { + comments.push({ id: comments.length + 1, body: (JSON.parse(body) as { body: string }).body }); + return send({ ...comments.at(-1), html_url: `https://github.example/pull/7#issuecomment-${comments.length}` }); + } + const id = Number(request.url?.split('/').at(-1)); + comments.find((comment) => comment.id === id)!.body = (JSON.parse(body) as { body: string }).body; + return send({ id, html_url: `https://github.example/pull/7#issuecomment-${id}` }); + }); + }); + await new Promise((done) => server.listen(0, '127.0.0.1', done)); + }); + after(() => server.close()); + + beforeEach(() => { + for (const name of VARIABLES) saved.set(name, process.env[name]); + for (const name of VARIABLES) delete process.env[name]; + comments.length = 0; + requests = []; + refuses = false; + scratch = mkdtempSync(join(tmpdir(), 'verify-github-')); + writeFileSync(join(scratch, 'event.json'), JSON.stringify({ issue: { number: 7, pull_request: {} } })); + writeFileSync(join(scratch, 'summary.md'), ''); + }); + afterEach(() => { + for (const [name, value] of saved) { + if (value === undefined) delete process.env[name]; + else process.env[name] = value; + } + }); + + const onActions = (event: string) => + Object.assign(process.env, { + GITHUB_ACTIONS: 'true', + GITHUB_REPOSITORY: 'clerk/unit', + GITHUB_RUN_ID: '1', + GITHUB_EVENT_NAME: event, + GITHUB_EVENT_PATH: join(scratch, 'event.json'), + GITHUB_WORKFLOW: 'PR CI', + GITHUB_JOB: 'e2e-tests', + GITHUB_API_URL: `http://127.0.0.1:${(server.address() as AddressInfo).port}`, + GITHUB_STEP_SUMMARY: join(scratch, 'summary.md'), + GITHUB_TOKEN: TOKEN, + }); + + const record = (tainted: readonly string[] = [], reports: readonly Report[] = GROUPS): EvidenceRecord => + ({ + platform: 'ios', + gitHead: '0123456789abcdef0123456789abcdef01234567', + tainted, + settings: reports.map((report, index) => { + const file = join(scratch, `e2e-${index}`, 'report.json') as EvidencePath; + mkdirSync(join(scratch, `e2e-${index}`), { recursive: true }); + writeFileSync(file, JSON.stringify(report)); + return { label: `group ${index + 1}`, e2eReport: file }; + }), + }) as unknown as EvidenceRecord; + + it('posts one comment for a run of three groups and updates that comment on the next run', async () => { + onActions('issue_comment'); + const first = await reportToGitHub({ record: record(), failures: [], packageDir: scratch }); + assert.deepEqual(first, ['https://github.example/pull/7#issuecomment-1']); + assert.deepEqual(requests, [`GET /repos/clerk/unit/issues/7/comments Bearer ${TOKEN}`, `POST /repos/clerk/unit/issues/7/comments Bearer ${TOKEN}`]); + const body = comments[0]!.body; + assert.equal(body.split('\n')[0], ''); + assert.match(body, /^### .* e2e ios: .*2 passed/m); + assert.match(body, /1 flaky test passed on a retry/); + assert.match(body, /the runner session ended/); + for (const row of ALL) assert.ok(body.includes(row.title), row.title); + assert.match(readFileSync(join(scratch, 'summary.md'), 'utf8'), /1 flaky test passed on a retry/); + + requests = []; + await reportToGitHub({ record: record(), failures: [], packageDir: scratch }); + assert.deepEqual(requests.map((line) => line.split(' ').slice(0, 2).join(' ')), ['GET /repos/clerk/unit/issues/7/comments', 'PATCH /repos/clerk/unit/issues/comments/1']); + assert.equal(comments.length, 1); + }); + + it('names a group that did not run in full, so a comment with fewer tests is not green', async () => { + onActions('issue_comment'); + await reportToGitHub({ record: record([], [GROUPS[0]]), failures: [{ label: 'MFA required', failure: new VerifyFailure('E2E_CRASHED', 'e2e exited 1 before writing a report', 'read e2e.log') }], packageDir: scratch }); + assert.match(comments[0]!.body, /E2E_CRASHED.* MFA required: e2e exited 1 before writing a report/); + assert.doesNotMatch(comments[0]!.body.split('\n')[1]!, /🟢/); + }); + + it('says why nothing was posted and does not throw when GitHub refuses the token', async () => { + onActions('issue_comment'); + refuses = true; + const lines = await reportToGitHub({ record: record(), failures: [], packageDir: scratch }); + assert.equal(lines.length, 1); + assert.match(lines[0]!, /^not reported: the token cannot comment on clerk\/unit#7/); + assert.equal(comments.length, 0); + }); + + it('writes the job summary and posts nothing when the event has no pull request', async () => { + onActions('workflow_dispatch'); + writeFileSync(join(scratch, 'event.json'), '{}'); + process.env.GITHUB_REF = 'refs/heads/main'; + const lines = await reportToGitHub({ record: record(), failures: [], packageDir: scratch }); + assert.deepEqual(lines, ['not posted: workflow_dispatch is not a pull request; written to the job summary']); + assert.deepEqual(requests, []); + assert.match(readFileSync(join(scratch, 'summary.md'), 'utf8'), /e2e ios/); + }); + + it('reports nothing for a run that holds a secret value, and nothing outside GitHub Actions', async () => { + onActions('issue_comment'); + assert.deepEqual(await reportToGitHub({ record: record(['e2e.log']), failures: [], packageDir: scratch }), ['not reported: a file of this run holds a secret value']); + assert.equal(readFileSync(join(scratch, 'summary.md'), 'utf8'), ''); + delete process.env.GITHUB_ACTIONS; + assert.deepEqual(await reportToGitHub({ record: record(), failures: [], packageDir: scratch }), ['not posted: not running on GitHub Actions']); + assert.deepEqual(requests, []); + }); + + it('treats the GitHub token of the environment as a secret, so output is redacted and the evidence scan looks for it', () => { + protectGitHubTokens({ GITHUB_TOKEN: TOKEN, GH_TOKEN: '' }); + assert.equal(redact(`authorization: Bearer ${TOKEN}`), 'authorization: Bearer '); + }); +}); diff --git a/integration/expo-native/test/host.test.ts b/integration/expo-native/test/host.test.ts new file mode 100644 index 00000000000..ab5c69e04f2 --- /dev/null +++ b/integration/expo-native/test/host.test.ts @@ -0,0 +1,202 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { describe, it } from 'node:test'; +import { existsSync, mkdtempSync, readFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { nativeInputs } from '../src/fixture.ts'; +import { host, metroPort, withCleanup } from '../src/host.ts'; +import { devClientEntry } from '../specs/app.ts'; +import { selectBackend } from '../src/core/devices.ts'; +import { resolveSpecs } from '../src/core/e2e.ts'; +import { STANDARD, declaredIn, planGroups, readSpecText } from '../src/core/instances/settings.ts'; +import { encodeLaunchArguments } from '../specs/support/launch.ts'; +import { VerifyFailure, type LaunchId, type PublishableKey, type RunId, type StorageScope } from '../src/core/types.ts'; +import { localAndroidBackend } from '../src/platform/android/local.ts'; +import { localIosBackend } from '../src/platform/ios/local.ts'; + +const WORKTREE = join(import.meta.dirname, '..', '..', '..'); + +describe('nativeInputs', () => { + it('rebuilds the dev client for the native code of its own platform and for the fixture config, and not for the other platform', () => { + for (const input of [ + 'packages/expo/ios', + 'packages/expo/app.plugin.js', + 'packages/expo/src/specs', + 'integration/templates/expo-native/app.json', + 'integration/templates/expo-native/modules', + ]) { + assert.ok(nativeInputs('ios').includes(input), input); + } + assert.ok(nativeInputs('android').includes('packages/expo/android')); + assert.deepEqual( + nativeInputs('ios').filter(input => input.endsWith('/android')), + [], + ); + assert.deepEqual( + nativeInputs('android').filter(input => input.endsWith('/ios')), + [], + ); + assert.deepEqual( + [...nativeInputs('ios'), ...nativeInputs('android')].filter( + input => input.startsWith('packages/expo/src/') && input !== 'packages/expo/src/specs', + ), + [], + ); + }); + + it('names only paths this repository has, so a moved directory cannot stop triggering a rebuild unseen', () => { + for (const input of new Set([...nativeInputs('ios'), ...nativeInputs('android')])) + assert.ok(existsSync(join(WORKTREE, input)), input); + }); +}); + +describe('the clerk-expo host', () => { + it('says what a machine without the device would need', () => { + const empty = mkdtempSync(join(tmpdir(), 'verify-expo-host-')); + const elsewhere = { + ...host, + backends: [ + localIosBackend({ os: 'linux' }), + localAndroidBackend({ machine: { os: 'linux', arch: 'x64', home: empty, env: {}, kvm: join(empty, 'kvm') } }), + ], + }; + assert.throws( + () => selectBackend(elsewhere, 'ios', undefined, null), + (error: VerifyFailure) => + error.code === 'UNSUPPORTED' && + /^no ios backend runs on this machine \(local: the iOS simulator needs macOS and this machine runs linux/.test( + error.message, + ) && + error.fix === 'run on a Mac with Xcode', + ); + assert.throws( + () => selectBackend(elsewhere, 'android', undefined, null), + (error: VerifyFailure) => + error.code === 'UNSUPPORTED' && + /^no android backend runs on this machine \(local: there is no .*kvm/.test(error.message), + ); + }); + + it('names the home links the fixture shows, and no other', () => { + const screens = join(WORKTREE, 'integration', 'templates', 'expo-native', 'screens'); + const homeLinkIds = (text: string): string[] => + [...text.matchAll(/'(e2e\.home\.[A-Za-z]+)'/g)].map(match => match[1] ?? '').sort(); + const shown = homeLinkIds( + ['Home.tsx', 'destinations.ts'].map(file => readFileSync(join(screens, file), 'utf8')).join('\n'), + ); + assert.ok(shown.length > 0); + assert.deepEqual(homeLinkIds(readFileSync(join(import.meta.dirname, '..', 'specs', 'native.ts'), 'utf8')), shown); + }); +}); + +describe('the golden specs of this repository', () => { + const dir = join(import.meta.dirname, '..'); + const golden = resolveSpecs(dir, { all: true }).map(spec => ({ spec, ...readSpecText(dir, spec.path) })); + + it('declare settings in one spec file, so a run of all of them is the standard group and then the biometric sign-in group', () => { + const declaring = golden + .filter(({ spec, ...text }) => declaredIn(text, spec.path) !== null) + .map(({ spec }) => spec.path); + assert.deepEqual(declaring, ['specs/golden/native-modules/biometric-availability.e2e.ts']); + const plan = planGroups(golden, STANDARD.key); + assert.equal(plan.length, 2); + assert.equal(plan[0]!.settings, STANDARD); + assert.equal(plan[0]!.specs.length, golden.length - 1); + assert.deepEqual( + plan[1]!.specs.map(spec => spec.path), + declaring, + ); + }); +}); + +describe('metroPort', () => { + it('gives every lane on the Mac its own port', () => { + const ports = [ + ...[1, 2, 3, 4].map(slot => metroPort({ platform: 'ios', slot })), + ...[1, 2].map(slot => metroPort({ platform: 'android', slot })), + ]; + assert.deepEqual(ports, [8082, 8083, 8084, 8085, 8086, 8087]); + }); +}); + +describe('devClientEntry', () => { + it('opens the iOS dev client on the lane Metro with the dev menu out of the way', () => { + const entry = devClientEntry('ios', 'http://localhost:8083'); + assert.equal(entry.kind, 'dev-client'); + if (entry.kind !== 'dev-client') return; + assert.deepEqual(entry.launchArguments.slice(0, 2), ['--initialUrl', 'http://localhost:8083']); + assert.ok(entry.launchArguments.includes('-EXDevMenuIsOnboardingFinished')); + assert.equal(entry.openLink, null); + }); + + it('opens the Android dev client through the exp+ link to the lane Metro', () => { + const entry = devClientEntry('android', 'http://localhost:8086'); + assert.equal(entry.kind, 'dev-client'); + if (entry.kind !== 'dev-client') return; + assert.equal( + entry.openLink, + 'exp+clerk-expo-native-build-fixture://expo-development-client/?url=http%3A%2F%2Flocalhost%3A8086', + ); + assert.deepEqual(entry.launchArguments, []); + }); + + it('keeps the iOS dev-client arguments compatible with the verify launch arguments', () => { + const verify = encodeLaunchArguments('ios', { + verifyPublishableKey: 'pk_test_x' as PublishableKey, + verifyRunId: 'r20261003-000000-abcd' as RunId, + verifyStorageScope: 'aa' as StorageScope, + verifyLaunchId: 'bb' as LaunchId, + }); + const entry = devClientEntry('ios', 'http://localhost:8082'); + if (entry.kind !== 'dev-client') return assert.fail('not a dev client'); + const keys = new Set([...entry.launchArguments, ...verify].filter(arg => arg.startsWith('-'))); + assert.equal( + keys.size, + entry.launchArguments.filter(a => a.startsWith('-')).length + verify.filter(a => a.startsWith('-')).length, + ); + }); +}); + +describe('withCleanup', () => { + it('stops exactly what the failed call started, then rethrows', async () => { + const stopped: string[][] = []; + await assert.rejects( + withCleanup( + names => stopped.push([...names]), + async started => { + started.names.push('watch', 'metro-8082'); + throw new Error('warm-up failed'); + }, + () => undefined, + ), + /warm-up failed/, + ); + assert.deepEqual(stopped, [['watch', 'metro-8082']]); + }); + + it('stops nothing when the call succeeds or started nothing', async () => { + const stopped: string[][] = []; + assert.equal( + await withCleanup( + names => stopped.push([...names]), + async started => { + started.names.push('watch'); + return 7; + }, + () => undefined, + ), + 7, + ); + await assert.rejects( + withCleanup( + names => stopped.push([...names]), + async () => { + throw new Error('refused'); + }, + () => undefined, + ), + ); + assert.deepEqual(stopped, []); + }); +}); diff --git a/integration/expo-native/test/inputs.test.ts b/integration/expo-native/test/inputs.test.ts new file mode 100644 index 00000000000..9aaa70c3a8f --- /dev/null +++ b/integration/expo-native/test/inputs.test.ts @@ -0,0 +1,140 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { pathToFileURL } from 'node:url'; +import { PACKAGE_DIR } from '../specs/support/config.ts'; +import { agentDevice } from '../specs/support/device.ts'; +import { INPUT_VARIABLES, inputsEnv, readClerk, readRun, readTarget, type Inputs } from '../specs/support/inputs.ts'; +import { Secret } from '../specs/support/secret.ts'; +import { BOTH_PLATFORMS_APP, IOS_APP, SAMPLE_INPUTS, SAMPLE_PUBLISHABLE_KEY, SAMPLE_RUN } from '../testing/sample-inputs.ts'; + +const UDID = 'FDF0CD9E-CF9E-42B6-AE3A-116A665F7EF3'; +const SECRET_KEY = 'sk_test_inputsUnitTestOnly'; + +describe('the device and app a run is told to use', () => { + it('needs only a device id in a repository with one platform', () => { + assert.deepEqual(readTarget(IOS_APP, { CLERK_E2E_DEVICE: UDID }), { + platform: 'ios', + device: { kind: 'local', id: UDID }, + session: 'clerk-e2e-ios', + build: { path: null, devServer: null }, + }); + }); + + it('asks which platform when the test app has two, and refuses one it does not have', () => { + assert.throws(() => readTarget(BOTH_PLATFORMS_APP, { CLERK_E2E_DEVICE: UDID }), /CLERK_E2E_PLATFORM is not set, and this test app runs on ios and android/); + assert.equal(readTarget(BOTH_PLATFORMS_APP, { CLERK_E2E_PLATFORM: 'android', CLERK_E2E_DEVICE: 'emulator-5554' }).platform, 'android'); + assert.throws(() => readTarget(BOTH_PLATFORMS_APP, { CLERK_E2E_PLATFORM: 'web', CLERK_E2E_DEVICE: UDID }), /CLERK_E2E_PLATFORM is web/); + assert.throws(() => readTarget(IOS_APP, { CLERK_E2E_PLATFORM: 'android', CLERK_E2E_DEVICE: 'emulator-5554' }), /this test app runs on ios$/); + }); + + it('names the command that prints device ids when none is given, and does not take a device name', () => { + assert.throws(() => readTarget(IOS_APP, {}), /CLERK_E2E_DEVICE is not set; .*`xcrun simctl list devices booted` prints/); + assert.throws(() => readTarget(IOS_APP, { CLERK_E2E_DEVICE: ' ' }), /CLERK_E2E_DEVICE is not set/); + assert.throws(() => readTarget(BOTH_PLATFORMS_APP, { CLERK_E2E_PLATFORM: 'android' }), /`adb devices` prints/); + assert.throws(() => readTarget(IOS_APP, { CLERK_E2E_DEVICE: 'iPhone 17 Pro' }), /a name is not accepted/); + assert.throws(() => readTarget(IOS_APP, { CLERK_E2E_DEVICE: 'booted' }), /a name is not accepted/); + for (const notASerial of ['Pixel 9', 'Pixel_9', 'Clerk_Verify_Pixel', '127.0.0.1:5555', 'R58M12ABCDE']) { + assert.throws(() => readTarget(BOTH_PLATFORMS_APP, { CLERK_E2E_PLATFORM: 'android', CLERK_E2E_DEVICE: notASerial }), /is not the emulator- serial of an emulator; an AVD name, a host:port address, and the serial of a phone are not accepted, so copy the id that `adb devices` prints/, notASerial); + } + assert.throws(() => readTarget(IOS_APP, { CLERK_E2E_DEVICE: '00008030-001A2B3C4D5E6F78' }), /is not the UDID of a simulator/, 'the id of a phone'); + }); + + it('takes an id only when the engine that drives the device reads it as that id, as the command that types on the device does', async () => { + const pool = (await import(pathToFileURL(join(PACKAGE_DIR, 'node_modules', '@e2e-dev', 'mobile', 'dist', 'pool.js')).href)) as { deviceSelection(platform: string, binding: { device: string }): Record }; + const typingSelects = (platform: 'ios' | 'android', id: string): Record => { + const [, , flag, value] = agentDevice({ kind: 'local', id }, platform).selector; + return { platform, [flag!.replace(/^--/, '')]: value! }; + }; + const ids = { ios: [UDID, UDID.toLowerCase(), '00008030-001A2B3C4D5E6F78', 'iPhone 17 Pro'], android: ['emulator-5554', 'emulator-5560', 'Pixel_9', 'Clerk_Verify_Pixel', '127.0.0.1:5555', 'R58M12ABCDE'] } as const; + for (const platform of ['ios', 'android'] as const) { + for (const id of ids[platform]) { + const sameDevice = JSON.stringify(pool.deviceSelection(platform, { device: id })) === JSON.stringify(typingSelects(platform, id)); + const accepted = ((): boolean => { + try { + return readTarget(BOTH_PLATFORMS_APP, { CLERK_E2E_PLATFORM: platform, CLERK_E2E_DEVICE: id }).device.id === id; + } catch { + return false; + } + })(); + assert.equal(accepted, sameDevice, `${id} on ${platform}: the engine selects ${JSON.stringify(pool.deviceSelection(platform, { device: id }))}`); + } + } + }); + + it('takes a build to install, a dev server, and a session name when they are given', () => { + const target = readTarget(BOTH_PLATFORMS_APP, { CLERK_E2E_PLATFORM: 'ios', CLERK_E2E_DEVICE: UDID, CLERK_E2E_APP_PATH: '/tmp/E2EHost.app', CLERK_E2E_DEV_SERVER: 'http://localhost:8081', CLERK_E2E_DEVICE_SESSION: 'mine' }); + assert.deepEqual([target.build, target.session], [{ path: '/tmp/E2EHost.app', devServer: 'http://localhost:8081' }, 'mine']); + }); +}); + +describe('the Clerk instance a run is told to use', () => { + const pk = { CLERK_PUBLISHABLE_KEY: SAMPLE_PUBLISHABLE_KEY }; + + it('reaches Clerk with the secret key of a development instance', () => { + const clerk = readClerk({ ...pk, CLERK_SECRET_KEY: SECRET_KEY }); + assert.equal(clerk.publishableKey, SAMPLE_PUBLISHABLE_KEY); + assert.equal(clerk.access.kind === 'secret-key' && clerk.access.key.use('bapi-authorization', (plain) => plain), SECRET_KEY); + assert.equal(JSON.stringify(clerk).includes(SECRET_KEY), false, 'the key does not print'); + }); + + it('or with a stand-in on this machine and its token file', () => { + assert.deepEqual(readClerk({ ...pk, CLERK_E2E_API_URL: 'http://127.0.0.1:4010/v1', CLERK_E2E_API_TOKEN_FILE: '/tmp/token' }).access, { kind: 'stand-in', url: 'http://127.0.0.1:4010/v1', tokenFile: '/tmp/token' }); + }); + + it('refuses a production instance, since the tests create and sign in users', () => { + assert.throws(() => readClerk({ CLERK_PUBLISHABLE_KEY: 'pk_live_abc', CLERK_SECRET_KEY: SECRET_KEY }), /CLERK_PUBLISHABLE_KEY is not a pk_test_ key/); + assert.throws(() => readClerk({ ...pk, CLERK_SECRET_KEY: 'sk_live_abc' }), /CLERK_SECRET_KEY is not an sk_test_ key/); + }); + + it('refuses a missing key, and two ways to reach Clerk at once', () => { + assert.throws(() => readClerk({ CLERK_SECRET_KEY: SECRET_KEY }), /CLERK_PUBLISHABLE_KEY is not set/); + assert.throws(() => readClerk(pk), /CLERK_SECRET_KEY is not set/); + assert.throws(() => readClerk({ ...pk, CLERK_SECRET_KEY: SECRET_KEY, CLERK_E2E_API_URL: 'http://127.0.0.1:4010/v1', CLERK_E2E_API_TOKEN_FILE: '/tmp/token' }), /are both set/); + }); + + it('sends its requests to no stand-in that is off this machine, and to none without a token', () => { + for (const url of ['https://api.example.com/v1', 'http://localhost:4010/v1', 'https://127.0.0.1:4010/v1', 'http://127.0.0.1.example.com:4010/v1', 'http://127.0.0.1/v1', 'http://user@127.0.0.1:4010/v1', 'http://127.0.0.1:1@evil.example/v1', 'http://127.0.0.1:4010.evil.example/v1', 'http://127.0.0.1:4010evil.example/v1']) { + assert.throws(() => readClerk({ ...pk, CLERK_E2E_API_URL: url, CLERK_E2E_API_TOKEN_FILE: '/tmp/token' }), /is not an http:\/\/127\.0\.0\.1: address/, url); + } + assert.throws(() => readClerk({ ...pk, CLERK_E2E_API_URL: 'http://127.0.0.1:4010/v1' }), /CLERK_E2E_API_TOKEN_FILE is not/); + }); +}); + +describe('the id of a run', () => { + it('is the one given, or a new one, and never something that is not a run id', () => { + assert.equal(readRun({ CLERK_E2E_RUN_ID: SAMPLE_RUN }), SAMPLE_RUN); + assert.match(readRun({}), /^r\d{8}-\d{6}-[0-9a-f]{4}$/); + assert.throws(() => readRun({ CLERK_E2E_RUN_ID: '../../etc' }), /is not a run id/); + }); +}); + +describe('the settings the CLI hands to e2e', () => { + const byHand: Inputs = { + target: { platform: 'android', device: { kind: 'local', id: 'emulator-5554' }, session: 'verify-android-abc', build: { path: '/builds/e2e-debug.apk', devServer: 'http://localhost:8085' } }, + clerk: { publishableKey: SAMPLE_PUBLISHABLE_KEY, access: { kind: 'secret-key', key: new Secret('clerk-secret-key', SECRET_KEY) } }, + run: SAMPLE_RUN, + }; + + it('read back as the same device, app build, instance and run', () => { + const env = inputsEnv(SAMPLE_INPUTS); + assert.deepEqual(readTarget(IOS_APP, env), SAMPLE_INPUTS.target); + assert.deepEqual(readClerk(env), SAMPLE_INPUTS.clerk); + assert.equal(readRun(env), SAMPLE_INPUTS.run); + + const handEnv = inputsEnv(byHand); + assert.deepEqual(readTarget(BOTH_PLATFORMS_APP, handEnv), byHand.target); + const clerk = readClerk(handEnv); + assert.equal(clerk.access.kind === 'secret-key' && clerk.access.key.use('bapi-authorization', (plain) => plain), SECRET_KEY); + }); + + it('carry no secret key when the CLI stands in for Clerk, and no app path the CLI did not name', () => { + const env = inputsEnv(SAMPLE_INPUTS); + assert.deepEqual(Object.keys(env).sort(), ['CLERK_E2E_API_TOKEN_FILE', 'CLERK_E2E_API_URL', 'CLERK_E2E_DEVICE', 'CLERK_E2E_DEVICE_SESSION', 'CLERK_E2E_PLATFORM', 'CLERK_E2E_RUN_ID', 'CLERK_PUBLISHABLE_KEY']); + }); + + it('are all named in the list the CLI clears from the environment it inherits', () => { + for (const name of [...Object.keys(inputsEnv(SAMPLE_INPUTS)), ...Object.keys(inputsEnv(byHand))]) assert.ok(INPUT_VARIABLES.includes(name), name); + }); +}); diff --git a/integration/expo-native/test/instances.test.ts b/integration/expo-native/test/instances.test.ts new file mode 100644 index 00000000000..22c5a093e85 --- /dev/null +++ b/integration/expo-native/test/instances.test.ts @@ -0,0 +1,1677 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, statSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { startBroker } from '../src/core/broker.ts'; +import { createClerkBackends } from '../src/core/clerk.ts'; +import { newTestEmail } from '../specs/support/clerk.ts'; +import { newRunId } from '../specs/support/inputs.ts'; +import type { ExecResult, Runner } from '../src/core/exec.ts'; +import { compareEnvironment, flattenEnvironment } from '../src/core/instances/settings.ts'; +import { createInstances } from '../src/core/instances/instances.ts'; +import { createPlatform, deadlineOf, describeCredential, throwawayApplication, throwawayName } from '../src/core/instances/platform.ts'; +import { STANDARD, STANDARD_ENVIRONMENT_KEY, STANDARD_FILE, SettingsRefused, planGroups, standardFile, type SettingsGroup } from '../src/core/instances/settings.ts'; +import { openApplications } from '../src/core/instances/throwaway.ts'; +import { withoutClerkKeys } from '../src/core/keys.ts'; +import { finishOrphanLedgers } from '../src/core/ledgers.ts'; +import { createOutput, takePlatformKey } from '../src/core/cli.ts'; +import { Secret, redact } from '../specs/support/secret.ts'; +import { down, type Deps } from '../src/core/verbs.ts'; +import { newEntryId, openWorkspace, takeSlotLock } from '../src/core/workspace.ts'; +import { RETRYABLE, VerifyFailure, type DeviceBackend, type HostAdapter, type InstanceSettings, type InstanceView, type LocalLease, type PublishableKey, type ScratchPath } from '../src/core/types.ts'; +import { PLATFORM_KEY, PLATFORM_SCOPES, SECRET_KEY_SCOPE, WORKSPACE, fakeClerk, type FakeClerkOptions } from '../testing/fake-clerk.ts'; + +const T0 = Date.parse('2026-10-05T12:00:00Z'); +const HOUR = 3_600_000; +const host = { repo: 'clerk-ios', platforms: ['ios'] } as unknown as HostAdapter; + +const MFA: InstanceSettings = { config: { auth_multi_factor: { required_for_sign_up: true } }, environment: { 'user_settings.sign_up.mfa.required': true } }; +const FORCED_ORG: InstanceSettings = { config: { organization_settings: { force_organization_selection: true } }, environment: { 'organization_settings.force_organization_selection': true } }; +const MFA_LABEL = 'auth_multi_factor.required_for_sign_up=true'; +const ORG_LABEL = 'organization_settings.force_organization_selection=true'; +const MFA_BODY = { ...standardFile().config, auth_multi_factor: { ...(standardFile().config.auth_multi_factor as object), required_for_sign_up: true } }; +const MFA_SPEC = 'specs/golden/session-tasks/setup-mfa.e2e.ts'; +const MFA_FILE = 'specs/golden/session-tasks/setup-mfa.settings.json'; +const ORG_SPEC = 'specs/golden/session-tasks/choose-organization.e2e.ts'; +const STANDARD_SPEC = 'specs/golden/auth-start/auth-start.e2e.ts'; + +const groupOf = (declared: InstanceSettings | null, path: string): SettingsGroup => + planGroups([{ spec: { kind: 'golden', path, feature: null }, source: "test('x', () => {});\n", declaration: declared === null ? null : JSON.stringify(declared) }], null)[0]!; +const keyRefused = (error: VerifyFailure): boolean => + error.code === 'NOT_READY' && + error.message === "api.clerk.com answered 401 to the instance's own secret key; the likely cause is a cloud environment whose API credential for api.clerk.com has no path prefix, so it replaces the key on every request to that host" && + error.fix === 'set Path prefixes on that credential to /v1/platform/, so it is attached to Platform API calls only, then rerun'; +const standardGroup = groupOf(null, STANDARD_SPEC); +const mfaGroup = groupOf(MFA, MFA_SPEC); +const orgGroup = groupOf(FORCED_ORG, ORG_SPEC); + +const idOf = (view: InstanceView | undefined): string | null => view?.id ?? null; +const quiet = () => undefined; +const NOTHING_NEW = { willChange: false } as const; + +const noOp: Runner = async () => ({ code: 127, stdout: '', stderr: 'spawn op ENOENT' }); +const REFERENCE = 'op://fake-vault/fake-item/credential'; +const REFERENCE_SHAPE = 'op:////credential'; +const namesTheItem = (text: string): boolean => [REFERENCE, 'fake-vault', 'fake-item'].some((part) => text.includes(part)); +const opAnswers = (read: ExecResult): Runner => async (_command, args) => (args[0] === '--version' ? { code: 0, stdout: '2.30.0', stderr: '' } : read); + +function world(options: { readonly env?: Record; readonly clerk?: FakeClerkOptions; readonly runner?: Runner; readonly dir?: string; readonly home?: string; readonly shared?: ReturnType } = {}) { + const dir = options.dir ?? mkdtempSync(join(tmpdir(), 'verify-instances-')); + const clock = { at: T0 }; + const clerk = options.shared ?? fakeClerk({ now: () => clock.at, ...options.clerk }); + const workspace = openWorkspace({ packageDir: dir, worktree: dir, home: options.home ?? join(dir, 'home') }); + const lines: string[] = []; + const slept: number[] = []; + const env = options.env ?? { CLERK_PLATFORM_API_KEY: PLATFORM_KEY }; + const alive = new Set(); + const open = (pid = 1000, fetchImpl: typeof fetch = clerk.fetch) => + createInstances({ + workspace, + env, + runner: options.runner ?? noOp, + progress: (line) => lines.push(line), + fetch: fetchImpl, + sleep: async (ms) => { + slept.push(ms); + clock.at += ms; + }, + now: () => clock.at, + drivers: { self: { pid, startedAt: T0 }, isRunning: (driver) => alive.has(driver.pid) }, + }); + const say = (line: string) => void lines.push(line); + return { dir, clock, clerk, workspace, lines, slept, env, alive, open, say, instances: open() }; +} + +type World = ReturnType; + +const LEAVES = Object.keys(standardFile().environment).length; + +const platformCalls = (w: World) => w.clerk.platformRequests().map((request) => `${request.method} ${request.url.replace('api.clerk.com/v1/platform', '').split('?')[0]!.replace(/app_\w+/, '{app}').replace(/ins_\w+/, '{ins}')}`); +const writes = (w: World) => w.clerk.platformRequests().filter((request) => request.method !== 'GET').map((request) => `${request.method} ${request.url.replace('api.clerk.com/v1/platform', '').replace(/\/instances\/ins_\w+/, '')}`); +const patches = (w: World) => w.clerk.platformRequests().filter((request) => request.method === 'PATCH'); +const instancesDir = (w: World) => join(w.workspace.root, 'instances'); +const stateOf = (w: World, name: string) => JSON.parse(readFileSync(join(instancesDir(w), `${name}.state.json`), 'utf8')) as { environmentKey?: string; settings?: { key: string; label: string; askedBy: string | null }; drivers: { pid: number }[] }; + +const sentSince = (w: World, before: number): string[] => w.clerk.platformRequests().slice(before).map((request) => `${request.method} ${request.url.replace('api.clerk.com/v1/platform', '')}`); +const secretKeyReads = (w: World): ReturnType => w.clerk.platformRequests().filter((request) => request.url.endsWith('?include_secret_keys=true')); +const readOf = (application: { readonly id: string }): string => `GET /applications/${application.id}?include_secret_keys=true`; + +function filesHolding(w: World, value: string): string[] { + const under = (dir: string): string[] => + readdirSync(dir, { withFileTypes: true }).flatMap((entry) => (entry.isDirectory() ? under(join(dir, entry.name)) : readFileSync(join(dir, entry.name), 'utf8').includes(value) ? [join(dir, entry.name)] : [])); + return under(w.dir).sort(); +} + +function plantedApplication(w: World, random: string, secretKey?: string) { + const application = w.clerk.plant(throwawayName(new Date(T0 + 6 * HOUR), random), secretKey); + w.workspace.append({ id: newEntryId(), kind: 'application', name: application.name, workspace: WORKSPACE }); + mkdirSync(instancesDir(w), { recursive: true }); + return application; +} + +function heldWithoutState(w: World, random: string, secretKey?: string) { + const application = plantedApplication(w, random, secretKey); + writeFileSync(join(instancesDir(w), `${application.name}.json`), `${JSON.stringify({ application: application.id, instanceId: application.instanceId, publishableKey: application.pk })}\n`, { mode: 0o600 }); + return application; +} + +describe('comparing a public environment with the standard file', () => { + it('pins the settings a spec cannot run without, which no config key sets', () => { + const { environment } = standardFile(); + for (const leaf of ['auth_config.test_mode', 'auth_config.native_settings.api_enabled', 'user_settings.attributes.ticket.enabled', 'user_settings.actions.create_organization']) assert.equal(environment[leaf], true, leaf); + }); + + it('reports each expected leaf that differs or is absent, and nothing about a leaf it was not asked for', () => { + const live = { auth_config: { test_mode: false, reverification: false, brand_new: 1 } }; + const compared = compareEnvironment({ 'auth_config.test_mode': true, 'auth_config.reverification': false, 'auth_config.single_session_mode': false, toString: true }, live); + assert.equal(compared.compared, 4); + assert.deepEqual(compared.differing, [ + { path: 'auth_config.test_mode', expected: true, found: false }, + { path: 'auth_config.single_session_mode', expected: false, found: undefined }, + { path: 'toString', expected: true, found: undefined }, + ]); + }); + + it('reads a list of plain values as one leaf whatever its order', () => { + assert.deepEqual(flattenEnvironment({ a: ['b', 'a'], c: [{ d: 1 }] }), { a: ['a', 'b'], 'c[0].d': 1 }); + }); +}); + +describe('throwaway names', () => { + it('carry the deadline, and only names this tool made parse', () => { + const name = throwawayName(new Date('2026-10-06T03:12:59Z'), '9c1f04ab'); + assert.equal(name, 'verify-throwaway-until-20261006t0312z-9c1f04ab'); + assert.equal(deadlineOf(name)?.toISOString(), '2026-10-06T03:12:00.000Z'); + assert.equal(deadlineOf('verify-throwaway-9c1f04ab77e2'), null); + assert.equal(deadlineOf('my-app-until-20261006t0312z-9c1f04ab'), null); + }); +}); + +describe('platform credential', () => { + const open = (env: Record, clerk: FakeClerkOptions = {}, runner: Runner = noOp) => { + const fake = fakeClerk(clerk); + const lines: string[] = []; + const ran: string[][] = []; + const platform = createPlatform({ env, runner: (command, args, options) => (ran.push([command, ...args]), runner(command, args, options)), progress: (line) => lines.push(line), fetch: fake.fetch, sleep: async () => undefined }); + return { fake, lines, ran, platform }; + }; + const op = opAnswers; + const referenced = { VERIFY_PLATFORM_KEY_REFERENCE: REFERENCE }; + const readsOf = (ran: readonly string[][]) => ran.filter((command) => command[1] === 'read'); + + it('uses the environment variable and asks 1Password nothing', async () => { + const { fake, ran, platform } = open({ CLERK_PLATFORM_API_KEY: PLATFORM_KEY }); + const workspace = await platform.open(); + assert.equal(workspace.credential.via, 'environment'); + assert.equal(workspace.workspace, WORKSPACE); + assert.deepEqual(fake.requests.map((r) => `${r.method} ${r.url}`), ['GET api.clerk.com/v1/platform/me', 'GET api.clerk.com/v1/platform/applications']); + assert.deepEqual(ran, []); + }); + + it('fails on a set variable that does not work, with no fall-through', async () => { + const { ran, platform } = open({ CLERK_PLATFORM_API_KEY: 'ak_someOtherKey' }, { attachesKey: false }); + await assert.rejects(platform.open(), (error: VerifyFailure) => error.code === 'KEYS_MISSING' && /CLERK_PLATFORM_API_KEY is set/.test(error.message)); + assert.deepEqual(ran, []); + }); + + it('reads the key from a private file, and refuses one other users can read', async () => { + const dir = mkdtempSync(join(tmpdir(), 'verify-key-')); + const file = join(dir, 'key'); + writeFileSync(file, `${PLATFORM_KEY}\n`, { mode: 0o600 }); + assert.equal((await open({ CLERK_PLATFORM_API_KEY_FILE: file }).platform.open()).credential.via, 'environment'); + chmodSync(file, 0o644); + await assert.rejects(open({ CLERK_PLATFORM_API_KEY_FILE: file }).platform.open(), (error: VerifyFailure) => error.fix === `chmod 600 ${file}`); + }); + + it('holds no key when something outside the machine attaches one', async () => { + const { fake, ran, platform } = open({}, { attachesKey: true }); + const workspace = await platform.open(); + assert.equal(workspace.credential.via, 'proxy'); + assert.ok(fake.requests.every((request) => request.authorization === null), 'this process sent no Authorization header'); + assert.deepEqual(ran, [], '1Password is not asked when the proxy already works'); + }); + + it('asks 1Password last, once, by reference, and never says what the reference is', async () => { + const { ran, lines, platform } = open(referenced, {}, op({ code: 0, stdout: PLATFORM_KEY, stderr: '' })); + const workspace = await platform.open(); + await platform.open(); + assert.deepEqual(Object.keys(workspace.credential).sort(), ['key', 'via']); + assert.equal(describeCredential(workspace.credential), '1Password'); + assert.deepEqual(readsOf(ran), [['op', 'read', '--no-newline', REFERENCE]], 'one read for the whole process'); + assert.equal(ran.flat().includes(PLATFORM_KEY), false, 'the key is never an argument'); + assert.deepEqual(lines, ['wait reading the team key from 1Password; approve the request in the 1Password app within 60s']); + }); + + it('prints no line that holds the reference when a command reads the key from 1Password', async () => { + const w = world({ env: referenced, runner: op({ code: 0, stdout: PLATFORM_KEY, stderr: '' }) }); + const up = await w.instances.ensure(NOTHING_NEW, w.say); + assert.ok(w.lines.includes(`instances 1Password reaches the verification workspace ${WORKSPACE}`)); + assert.ok(w.lines.length > 1); + assert.deepEqual([...w.lines, JSON.stringify(up)].filter(namesTheItem), []); + }); + + it('never runs op when no reference is set, and says how to set one without naming the item', async () => { + const unset: readonly Record[] = [{}, { VERIFY_PLATFORM_KEY_REFERENCE: ' ' }]; + for (const env of unset) { + const { ran, platform } = open(env, {}, op({ code: 0, stdout: PLATFORM_KEY, stderr: '' })); + await assert.rejects(platform.open(), (error: VerifyFailure) => { + assert.equal(error.code, 'KEYS_MISSING'); + assert.equal(error.message, 'no Clerk Platform API credential works here (CLERK_PLATFORM_API_KEY and CLERK_PLATFORM_API_KEY_FILE are not set; a request with no key gets 401 authorization_header_format_invalid, so nothing outside this machine attaches one; no 1Password reference is set)'); + assert.equal( + error.fix, + `on a Mac, with the 1Password CLI installed and its desktop app integration on, set VERIFY_PLATFORM_KEY_REFERENCE to the 1Password secret reference of the item that holds the team key, of the shape ${REFERENCE_SHAPE}; in a cloud environment, add an API credential for api.clerk.com with path prefix /v1/platform/; anywhere, set CLERK_PLATFORM_API_KEY to the team key, or CLERK_PLATFORM_API_KEY_FILE to a file that holds it and that only you can read`, + ); + return true; + }); + assert.deepEqual(ran, []); + } + }); + + it('refuses a reference that is not an op:// reference, naming where it was and not what it was', async () => { + const { ran, platform } = open({ VERIFY_PLATFORM_KEY_REFERENCE: 'Fake Vault/fake item/credential' }, {}, op({ code: 0, stdout: PLATFORM_KEY, stderr: '' })); + await assert.rejects(platform.open(), (error: VerifyFailure) => { + assert.equal(error.code, 'USAGE'); + assert.equal(error.message, 'VERIFY_PLATFORM_KEY_REFERENCE does not hold a 1Password secret reference'); + assert.equal(error.fix, `put a reference of the shape ${REFERENCE_SHAPE} there, or remove it`); + return true; + }); + assert.deepEqual(ran, []); + }); + + it('does not fail over a malformed reference when a source it tries before 1Password works', async () => { + const malformed = { VERIFY_PLATFORM_KEY_REFERENCE: 'Fake Vault/fake item/credential' }; + const keyed = open({ ...malformed, CLERK_PLATFORM_API_KEY: PLATFORM_KEY }); + assert.equal((await keyed.platform.open()).credential.via, 'environment'); + const proxied = open(malformed, { attachesKey: true }); + assert.equal((await proxied.platform.open()).credential.via, 'proxy'); + assert.deepEqual([...keyed.ran, ...proxied.ran], []); + }); + + it('treats a missing op as no credential, and an unapproved read as an error', async () => { + await assert.rejects(open(referenced).platform.open(), (error: VerifyFailure) => error.code === 'KEYS_MISSING' && /op\) is not installed/.test(error.message) && error.fix.startsWith('on a Mac, install the 1Password CLI and turn on its desktop app integration; ') && !namesTheItem(error.fix)); + await assert.rejects(open(referenced, {}, op({ code: 124, stdout: '', stderr: '' })).platform.open(), (error: VerifyFailure) => error.message === 'op read of the 1Password item the reference names was not approved within 60s'); + await assert.rejects(open(referenced, {}, op({ code: 1, stdout: '', stderr: '[ERROR] authorization denied\nmore' })).platform.open(), (error: VerifyFailure) => error.message.endsWith('failed: [ERROR] authorization denied')); + }); + + it('does not print the reference, the vault, or the item when op repeats them in its error', async () => { + const reference = 'op://vault-only-here/item-only-here/credential'; + const stderr = `[ERROR] could not read secret '${reference}': "item-only-here" isn't an item in the "vault-only-here" vault\n`; + const { platform } = open({ VERIFY_PLATFORM_KEY_REFERENCE: reference }, {}, op({ code: 1, stdout: '', stderr })); + const failure = await platform.open().then(() => assert.fail('the read failed'), (error: VerifyFailure) => error); + assert.equal(failure.message, 'op read of the 1Password item the reference names failed: [ERROR] could not read secret \'\': "" isn\'t an item in the "" vault'); + let printed = ''; + const sink = { write: (text: string) => ((printed += text), true) }; + const output = createOutput(false, '/tmp', 'bin/control-x', sink, sink); + output.failure(failure); + output.progress(`op said ${stderr}`); + assert.equal(printed.includes(reference), false, 'the reference is a secret to every line the CLI prints'); + assert.match(printed, /op said \[ERROR\] could not read secret ''/); + }); + + it('calls a key from 1Password that fails the key in the item the reference names', async () => { + const elsewhere = open(referenced, { workspace: 'org_someRealWorkspace' }, op({ code: 0, stdout: PLATFORM_KEY, stderr: '' })); + await assert.rejects(elsewhere.platform.open(), (error: VerifyFailure) => error.message.startsWith('the key in the 1Password item the reference names belongs to workspace org_someRealWorkspace') && error.fix === 'use the team key; a key of any other workspace is refused'); + const stale = open(referenced, {}, op({ code: 0, stdout: 'ak_noLongerWorks', stderr: '' })); + await assert.rejects(stale.platform.open(), (error: VerifyFailure) => error.message === "Clerk's Platform API answered 401 could_not_authenticate_request to the key in the 1Password item the reference names" && !namesTheItem(`${error.message} ${error.fix}`)); + }); + + it('refuses a key of any other workspace before it lists or creates anything', async () => { + const { fake, platform } = open({ CLERK_PLATFORM_API_KEY: PLATFORM_KEY }, { workspace: 'org_someRealWorkspace' }); + await assert.rejects(platform.open(), (error: VerifyFailure) => error.code === 'KEYS_MISSING' && error.message.includes('org_someRealWorkspace') && error.message.includes(WORKSPACE)); + assert.deepEqual(fake.requests.map((r) => r.url), ['api.clerk.com/v1/platform/me']); + }); + + it('refuses a workspace that holds an application without the prefix', async () => { + const { fake, platform } = open({ CLERK_PLATFORM_API_KEY: PLATFORM_KEY }); + fake.plant('Production Dashboard'); + await assert.rejects(platform.open(), (error: VerifyFailure) => error.code === 'NOT_READY' && /does not start with verify-throwaway-/.test(error.message) && !error.message.includes('Production Dashboard')); + assert.ok(fake.requests.every((request) => request.method === 'GET')); + }); + + it('checks the workspace again before a create that comes long after the last check', async () => { + let at = T0; + const fake = fakeClerk({ now: () => at }); + const platform = createPlatform({ env: { CLERK_PLATFORM_API_KEY: PLATFORM_KEY }, runner: noOp, progress: () => undefined, fetch: fake.fetch, now: () => at }); + const workspace = await platform.open(); + await workspace.create(throwawayName(new Date(T0 + HOUR), 'aaaaaaaa')); + assert.equal(fake.requests.length, 3, 'a create right after the check adds one request'); + at += 60_000; + fake.plant('Someone Else'); + await assert.rejects(workspace.create(throwawayName(new Date(T0 + HOUR), 'bbbbbbbb')), /does not start with verify-throwaway-/); + assert.equal(fake.live().length, 2, 'nothing was created after the workspace changed'); + }); + + it('waits out a rate limit for as long as Clerk asks, then succeeds', async () => { + const fake = fakeClerk(); + const waits: number[] = []; + const lines: string[] = []; + const platform = createPlatform({ env: { CLERK_PLATFORM_API_KEY: PLATFORM_KEY }, runner: noOp, progress: (line) => lines.push(line), fetch: fake.fetch, sleep: async (ms) => void waits.push(ms) }); + fake.rateLimit(2, '7'); + await platform.open(); + assert.deepEqual(waits, [7000, 7000]); + assert.equal(lines.filter((line) => line.startsWith('wait Clerk\'s Platform API is rate limiting')).length, 2); + fake.rateLimit(1); + waits.length = 0; + await (await platform.open()).list(); + assert.deepEqual(waits, [2000], 'with no Retry-After it backs off on its own schedule'); + fake.rateLimit(1, '3600'); + waits.length = 0; + await (await platform.open()).list(); + assert.deepEqual(waits, [60_000], 'and never sleeps longer than a minute on Clerk\'s word'); + }); + + it('gives up on a rate limit that does not lift, with an error a caller may retry', async () => { + const fake = fakeClerk(); + const platform = createPlatform({ env: { CLERK_PLATFORM_API_KEY: PLATFORM_KEY }, runner: noOp, progress: () => undefined, fetch: fake.fetch, sleep: async () => undefined }); + fake.rateLimit(50); + await assert.rejects(platform.open(), (error: VerifyFailure) => error.code === 'RATE_LIMITED' && RETRYABLE.has(error.code)); + assert.ok(fake.requests.length > 1 && fake.requests.length <= 10, `it retried a bounded number of times (${fake.requests.length} requests)`); + }); + + it('refuses a list that is not one array, because a partial list would hide what the workspace holds', async () => { + for (const shape of ['envelope', 'unreadable'] as const) { + const { fake, platform } = open({ CLERK_PLATFORM_API_KEY: PLATFORM_KEY }); + fake.plant(throwawayName(new Date(T0), 'aaaaaaaa')); + fake.plant('Production Dashboard'); + fake.state.listShape = shape; + fake.state.pageSize = 1; + await assert.rejects(platform.open(), /no longer has a shape this tool can read in full/, shape); + assert.ok(fake.requests.every((request) => request.method === 'GET')); + } + }); + + it('checks the workspace again before a delete that comes long after the last check', async () => { + let at = T0; + const fake = fakeClerk({ now: () => at }); + const platform = createPlatform({ env: { CLERK_PLATFORM_API_KEY: PLATFORM_KEY }, runner: noOp, progress: () => undefined, fetch: fake.fetch, now: () => at }); + const workspace = await platform.open(); + const created = await workspace.create(throwawayName(new Date(T0 + HOUR), 'aaaaaaaa')); + at += 60_000; + fake.plant('Someone Else'); + await assert.rejects(workspace.delete(created.application), /does not start with verify-throwaway-/); + assert.equal(fake.live().length, 2, 'nothing was deleted after the workspace changed'); + }); + + it('cannot be asked to delete a name without the prefix', () => { + assert.throws(() => throwawayApplication('app_1', 'Production Dashboard'), /lacks the verify-throwaway- prefix/); + }); +}); + +describe('what Clerk says to a config change', () => { + const opened = async () => { + const fake = fakeClerk(); + const platform = createPlatform({ env: { CLERK_PLATFORM_API_KEY: PLATFORM_KEY }, runner: noOp, progress: quiet, fetch: fake.fetch, sleep: async () => undefined }); + const workspace = await platform.open(); + const created = await workspace.create(throwawayName(new Date(T0 + HOUR), 'aaaaaaaa')); + return { fake, workspace, created }; + }; + + it('answers with the whole object of each key the body named, and a dry run changes nothing', async () => { + const { fake, workspace, created } = await opened(); + const dry = await workspace.configure(created, MFA.config, { dryRun: true }); + assert.deepEqual(dry.after, { auth_multi_factor: { required_for_sign_up: true } }); + assert.equal(fake.live()[0]!.environment['user_settings.sign_up.mfa.required'], false); + await workspace.configure(created, standardFile().config); + const real = await workspace.configure(created, MFA.config); + assert.deepEqual(real.after, { auth_multi_factor: { ...(standardFile().config.auth_multi_factor as object), required_for_sign_up: true } }, 'a PATCH merges into what the instance has'); + assert.equal(fake.live()[0]!.environment['user_settings.sign_up.mfa.required'], true); + }); + + it('turns a refused body into a refusal that names the key when Clerk names it', async () => { + const { fake, workspace, created } = await opened(); + fake.state.refusals.push({ path: 'auth_email.no_such_toggle', status: 400, code: 'unknown_config_key', param: 'auth_email.no_such_toggle', message: 'is not a config key' }); + await assert.rejects(workspace.configure(created, { auth_email: { no_such_toggle: true } }), (error: SettingsRefused) => error instanceof SettingsRefused && error.code === 'INSTANCE_MISCONFIGURED' && error.param === 'auth_email.no_such_toggle' && error.said === 'auth_email.no_such_toggle (400 unknown_config_key): is not a config key'); + fake.state.refusals.push({ path: 'auth_attack_protection.pii_protection_enabled', value: false, status: 409, code: 'user_settings_invalid', message: 'the settings are not valid together' }); + await assert.rejects(workspace.configure(created, { auth_attack_protection: { pii_protection_enabled: false } }), (error: SettingsRefused) => error instanceof SettingsRefused && error.param === null && error.said === '409 user_settings_invalid: the settings are not valid together'); + fake.state.refusals.push({ path: 'compliance.legal_consent.enabled', value: true, status: 422, code: 'form_param_missing', param: ['terms_of_service_url', 'privacy_policy_url'], message: 'is required' }); + await assert.rejects(workspace.configure(created, { compliance: { legal_consent: { enabled: true } } }), (error: SettingsRefused) => error instanceof SettingsRefused && error.param === 'terms_of_service_url' && error.said === 'terms_of_service_url (422 form_param_missing): is required', 'any 4xx that is about the body is a refusal, and a code two errors share is said once'); + assert.deepEqual(fake.live()[0]!.config, {}, 'a refused body applied nothing'); + }); + + it('reads a 4xx that names a parameter as a refusal of the body whatever its status, except a rate limit', async () => { + const { fake, workspace, created } = await opened(); + for (const status of [401, 403, 404, 408]) { + fake.state.refusals = [{ path: 'auth_email.no_such_toggle', status, code: 'form_param_unknown', param: 'no_such_toggle', message: 'is unknown' }]; + await assert.rejects(workspace.configure(created, { auth_email: { no_such_toggle: true } }), (error: SettingsRefused) => error instanceof SettingsRefused && error.param === 'no_such_toggle' && error.said === `no_such_toggle (${status} form_param_unknown): is unknown`, String(status)); + fake.state.refusals = [{ path: 'auth_email.no_such_toggle', status, code: 'not_about_the_body', message: 'names no parameter' }]; + await assert.rejects(workspace.configure(created, { auth_email: { no_such_toggle: true } }), (error: VerifyFailure) => !(error instanceof SettingsRefused) && error.code === 'NOT_READY' && error.message.includes(`answered ${status} not_about_the_body`), String(status)); + } + fake.state.refusals = [{ path: 'auth_email.no_such_toggle', status: 429, code: 'too_many_requests', param: 'no_such_toggle', message: 'slow down' }]; + await assert.rejects(workspace.configure(created, { auth_email: { no_such_toggle: true } }), (error: VerifyFailure) => !(error instanceof SettingsRefused) && error.code === 'RATE_LIMITED'); + }); + + it('keeps an outage, a rate limit, and a credential problem what they were', async () => { + const { fake, workspace, created } = await opened(); + fake.state.failConfigure = 1; + await assert.rejects(workspace.configure(created, MFA.config), (error: VerifyFailure) => !(error instanceof SettingsRefused) && error.code === 'NOT_READY' && /answered 500/.test(error.message)); + fake.rateLimit(50); + await assert.rejects(workspace.configure(created, MFA.config), (error: VerifyFailure) => !(error instanceof SettingsRefused) && error.code === 'RATE_LIMITED'); + fake.rateLimit(0); + fake.live()[0]!.deleted = true; + await assert.rejects(workspace.configure(created, MFA.config), (error: VerifyFailure) => !(error instanceof SettingsRefused) && /answered 404/.test(error.message), 'an application that is gone is not a refused declaration'); + }); +}); + +describe('one throwaway application', () => { + it('is created by up, put on the whole standard config with one PATCH, and recorded', async () => { + const w = world(); + const up = await w.instances.ensure(NOTHING_NEW, w.say); + const application = w.clerk.live()[0]!; + assert.deepEqual(up, [{ id: application.id, name: application.name, created: true, settings: 'standard' }]); + assert.deepEqual(platformCalls(w), ['GET /me', 'GET /applications', 'POST /applications', 'PATCH /applications/{app}/instances/{ins}/config']); + assert.deepEqual(patches(w)[0]!.body, standardFile().config); + assert.equal(deadlineOf(application.name)?.getTime(), T0 + 6 * HOUR); + assert.ok(w.lines.includes(`instance creating ${application.name} in ${WORKSPACE}`)); + assert.ok(w.lines.includes(`instance ${application.id} up in 0.0s on standard, ${LEAVES} settings match ${STANDARD_FILE}`), w.lines.join('\n')); + assert.deepEqual(stateOf(w, application.name), { environmentKey: STANDARD_ENVIRONMENT_KEY, settings: STANDARD, drivers: [] }); + assert.equal(w.instances.recordedKey(), STANDARD.key); + }); + + it('asks the Platform API only for the secret key when it is already up, and says what it is on', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const before = w.clerk.platformRequests().length; + const again = await w.open().ensure(NOTHING_NEW, w.say); + assert.deepEqual(sentSince(w, before), ['GET /me', readOf(w.clerk.live()[0]!)], 'the credential is checked, the key is read, and the applications are not listed'); + assert.deepEqual(again.map((view) => [view.created, view.settings]), [[false, 'standard']]); + assert.ok(w.lines.includes('instances this worktree already holds throwaway instances')); + }); + + it('opens the credential when the run will change settings, though nothing needs creating', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const before = w.clerk.platformRequests().length; + await w.open().ensure({ willChange: true }, quiet); + assert.deepEqual(sentSince(w, before), ['GET /me', readOf(w.clerk.live()[0]!), 'GET /applications'], 'a missing credential or a 1Password prompt would have come here, before any device is leased'); + const noCredential = (): ReturnType => createInstances({ workspace: w.workspace, env: {}, runner: noOp, progress: quiet, fetch: w.clerk.fetch }); + const sent = w.clerk.requests.length; + await assert.rejects(noCredential().access(), (error: VerifyFailure) => error.code === 'KEYS_MISSING', 'the secret key is on no disk, so a held application needs the credential too, and up and run ask for it before they lease a device'); + for (const options of [{ willChange: true }, NOTHING_NEW]) await assert.rejects(noCredential().ensure(options, quiet), (error: VerifyFailure) => error.code === 'KEYS_MISSING'); + assert.deepEqual(w.clerk.requests.slice(sent).map((request) => `${request.authorization === null ? 'no key' : 'a key'} ${request.method} ${request.url}`), Array.from({ length: 3 }, () => 'no key GET api.clerk.com/v1/platform/me'), 'nothing but the question of whether a key is attached outside this machine'); + }); + + it('writes the name to the ledger before the create call, so a lost answer is still deleted', async () => { + const w = world(); + w.clerk.state.loseCreateAnswer = true; + await assert.rejects(w.instances.ensure(NOTHING_NEW, quiet), /answered 500/); + const [entry] = openApplications(w.workspace); + assert.equal(w.clerk.live()[0]?.name, entry?.name, 'Clerk has the application and the ledger has its name'); + assert.equal('instance' in (entry ?? {}), false, 'a new entry names no instance'); + w.clerk.state.loseCreateAnswer = false; + const finished = await w.open().finish(w.workspace, { keepApplications: false }, quiet); + assert.deepEqual(finished, [{ name: entry?.name }]); + assert.equal(w.clerk.live().length, 0); + assert.deepEqual(openApplications(w.workspace), []); + }); + + it('replaces an application whose keys were lost instead of leaking it', async () => { + const w = world(); + w.clerk.state.loseCreateAnswer = true; + await assert.rejects(w.instances.ensure(NOTHING_NEW, quiet)); + w.clerk.state.loseCreateAnswer = false; + const lost = w.clerk.live()[0]!.id; + const up = await w.open().ensure(NOTHING_NEW, w.say); + assert.deepEqual(w.clerk.live().map((a) => a.id), [idOf(up[0])]); + assert.notEqual(idOf(up[0]), lost); + assert.equal(openApplications(w.workspace).length, 1); + assert.ok(w.lines.some((line) => line.includes('retired (its keys are lost)'))); + }); + + it('recovers from a crash between create and configure by configuring the same application again', async () => { + const w = world(); + w.clerk.state.failConfigure = 1; + await assert.rejects(w.instances.ensure(NOTHING_NEW, quiet), /configuring verify-throwaway-/); + const created = w.clerk.live()[0]!; + assert.equal(created.environment['user_settings.password_settings.min_length'], 15, 'still as Clerk made it'); + assert.equal(w.open().recordedKey(), null); + const up = await w.open().ensure(NOTHING_NEW, w.say); + assert.deepEqual(up.map((view) => [idOf(view), view.created]), [[created.id, false]]); + assert.equal(w.clerk.applications.length, 1, 'no second application'); + assert.equal(created.environment['user_settings.password_settings.min_length'], 8); + assert.ok(w.lines.includes(`settings changing ${created.id} from unknown settings to standard, because what it is on is not recorded`)); + assert.ok(w.lines.some((line) => line.startsWith(`settings standard on ${created.id} in 0.0s (Clerk answered in 0.00s, the instance showed it 0.00s later), ${LEAVES} settings match`))); + }); + + it('fails when a required setting does not show after the PATCH', async () => { + const w = world(); + w.clerk.state.ignoreConfigKey = 'auth_config.single_session_mode'; + await assert.rejects(w.instances.ensure(NOTHING_NEW, quiet), (error: VerifyFailure) => error.code === 'INSTANCE_MISCONFIGURED' && !(error instanceof SettingsRefused) && error.message.includes(`does not match ${STANDARD_FILE}`) && error.message.includes('auth_config.single_session_mode is true, and the file says false')); + assert.ok(w.slept.length > 0, 'it waited for the setting to take effect before failing'); + assert.equal(openApplications(w.workspace).length, 1, 'the application stays in the ledger for down'); + assert.equal(w.open().recordedKey(), null, 'and nothing records it as being on the standard settings'); + }); + + it('compares the required settings and the declared ones, and nothing else the instance shows', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, w.say); + const application = w.clerk.live()[0]!; + assert.ok(w.lines.includes(`instance ${application.id} up in 0.0s on standard, ${LEAVES} settings match ${STANDARD_FILE}`)); + application.environment['auth_config.reverification'] = false; + const applied = await w.instances.apply(mfaGroup, w.say); + assert.equal(await applied.stillApplied(), true, 'a setting that nothing requires and no spec declared does not fail a run'); + await applied.release(); + assert.equal((await w.open().ensure(NOTHING_NEW, quiet)).length, 1); + assert.equal(patches(w).length, 2, 'and it causes no repair'); + assert.ok(w.lines.some((line) => line.startsWith(`settings ${MFA_LABEL} on ${application.id}`) && line.endsWith(`${LEAVES} settings match`))); + }); + + it('compares a leaf a declaration names though the standard file does not list it', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const application = w.clerk.live()[0]!; + w.clerk.state.alsoMoves = { 'auth_config.reverification': false }; + const declared: InstanceSettings = { config: MFA.config, environment: { ...MFA.environment, 'auth_config.reverification': false } }; + const applied = await w.instances.apply(groupOf(declared, MFA_SPEC), w.say); + assert.ok(w.lines.some((line) => line.startsWith(`settings ${MFA_LABEL} on ${application.id}`) && line.endsWith(`${LEAVES + 1} settings match`)), 'the declared leaf is one of the settings compared'); + assert.equal(await applied.stillApplied(), true); + application.environment['auth_config.reverification'] = true; + assert.equal(await applied.stillApplied(), false); + }); + + it('never writes a keys file over one that is there', async () => { + const w = world(); + const first = 'the first write\n'; + let cachedKeys = ''; + const writeFirst = (line: string) => { + const creating = /^instance creating (\S+) in /.exec(line); + if (creating === null) return; + cachedKeys = join(instancesDir(w), `${creating[1]}.json`); + mkdirSync(instancesDir(w), { recursive: true }); + writeFileSync(cachedKeys, first); + }; + await assert.rejects(w.instances.ensure(NOTHING_NEW, writeFirst), { code: 'EEXIST' }); + assert.equal(readFileSync(cachedKeys, 'utf8'), first); + assert.deepEqual(filesHolding(w, w.clerk.live()[0]!.sk), [], 'and the secret key of the application it could not record is on no disk'); + }); + + it('creates a new application when Clerk no longer serves the old one', async () => { + const w = world(); + const first = await w.instances.ensure(NOTHING_NEW, quiet); + w.clerk.live()[0]!.deleted = true; + const second = await w.open().ensure(NOTHING_NEW, w.say); + assert.notEqual(idOf(second[0]), idOf(first[0])); + assert.equal(openApplications(w.workspace).length, 1); + assert.ok(w.lines.includes(`instance ${idOf(first[0])} retired (Clerk no longer serves it)`)); + }); + + it('replaces an application whose deadline is near, before a run starts on it', async () => { + const w = world(); + const first = await w.instances.ensure(NOTHING_NEW, quiet); + w.clock.at = T0 + 6 * HOUR - 20 * 60_000; + const second = await w.open().ensure(NOTHING_NEW, w.say); + assert.notEqual(idOf(second[0]), idOf(first[0])); + assert.deepEqual(w.clerk.live().map((a) => a.id), [idOf(second[0])], 'the old one was deleted, not left for the reaper'); + assert.ok(w.lines.some((line) => line.includes('its deadline is near'))); + }); + + it('replaces an application that holds 60 of the 100 users a development instance allows', async () => { + const w = world(); + const first = await w.instances.ensure(NOTHING_NEW, quiet); + w.clerk.live()[0]!.users = 59; + assert.equal(idOf((await w.open().ensure(NOTHING_NEW, quiet))[0]), idOf(first[0]), '59 users is still room for a run'); + w.clerk.live()[0]!.users = 60; + const second = await w.open().ensure(NOTHING_NEW, w.say); + assert.notEqual(idOf(second[0]), idOf(first[0])); + assert.deepEqual(w.clerk.live().map((a) => a.id), [idOf(second[0])]); + assert.ok(w.lines.includes(`instance ${idOf(first[0])} retired (it holds 60 of the 100 users a development instance allows)`)); + }); + + it('deletes an application it retires by its id when the listing it holds was taken before the application existed', async () => { + const w = world(); + const first = await w.instances.ensure(NOTHING_NEW, quiet); + w.clerk.live()[0]!.users = 60; + const stale = (async (input: string | URL, init?: RequestInit) => ((init?.method ?? 'GET') === 'GET' && String(input).endsWith('/platform/applications') ? new Response('[]', { status: 200 }) : w.clerk.fetch(input, init))) as typeof fetch; + const second = await w.open(1000, stale).ensure(NOTHING_NEW, w.say); + assert.ok(w.lines.includes(`instance ${idOf(first[0])} retired (it holds 60 of the 100 users a development instance allows)`)); + assert.deepEqual(w.clerk.live().map((a) => a.id), [idOf(second[0])], 'the retired application is gone from Clerk, not only from the ledger'); + }); + + it('does not treat a Frontend API error as a deleted application', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const broken = (async (input: string | URL, init?: RequestInit) => (String(input).includes('.clerk.accounts.dev') ? new Response('bad gateway', { status: 502 }) : w.clerk.fetch(input, init))) as typeof fetch; + await assert.rejects(w.open(1000, broken).ensure(NOTHING_NEW, quiet), /answered 502/); + assert.equal(w.clerk.live().length, 1); + }); + + it('sets the deadline by Clerk\'s clock when this machine\'s clock is wrong', async () => { + const dir = mkdtempSync(join(tmpdir(), 'verify-instances-')); + const clerk = fakeClerk({ now: () => T0 }); + const instances = createInstances({ workspace: openWorkspace({ packageDir: dir, worktree: dir, home: join(dir, 'home') }), env: { CLERK_PLATFORM_API_KEY: PLATFORM_KEY }, runner: noOp, progress: quiet, fetch: clerk.fetch, now: () => T0 - 5 * HOUR }); + await instances.ensure(NOTHING_NEW, quiet); + assert.equal(deadlineOf(clerk.live()[0]!.name)?.getTime(), T0 + 6 * HOUR); + }); + + it('fails with the likely cause and the fix when api.clerk.com refuses the own key of a new application, asks once, and keeps the application for down', async () => { + const w = world(); + w.clerk.state.refuseInstanceKeys = true; + await assert.rejects(w.instances.ensure(NOTHING_NEW, quiet), keyRefused); + assert.equal(openApplications(w.workspace).length, 1, 'the application stays in the ledger for down'); + assert.deepEqual(w.clerk.requests.filter((request) => request.url.startsWith('api.clerk.') && !request.url.startsWith('api.clerk.com/v1/platform/')).map((request) => request.url), ['api.clerk.com/v1/users/count'], 'one call, to api.clerk.com, and none to another name of the API'); + }); + + it('creates no application to apply a group, because only up and the start of a run create one', async () => { + const w = world(); + await assert.rejects(w.instances.apply(mfaGroup, quiet), (error: VerifyFailure) => error.code === 'NOT_READY' && error.message === 'this worktree holds no application to put on the settings of a group' && error.fix === '{cli} up'); + assert.deepEqual(w.clerk.requests, []); + }); + + it('rejects a lifetime shorter than a session can last', async () => { + const w = world({ env: { CLERK_PLATFORM_API_KEY: PLATFORM_KEY, VERIFY_THROWAWAY_HOURS: '0' } }); + await assert.rejects(w.instances.ensure(NOTHING_NEW, quiet), (error: VerifyFailure) => error.code === 'USAGE'); + assert.equal(w.clerk.applications.length, 0); + }); + + it('serves keys only while an instance is applied', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + assert.throws(() => w.instances.keys(), (error: VerifyFailure) => error.code === 'NOT_READY' && /no instance is applied/.test(error.message)); + assert.throws(() => w.instances.clerk(), /no instance is applied/); + const applied = await w.instances.apply(standardGroup, quiet); + assert.equal(w.instances.keys().pk, w.clerk.live()[0]!.pk); + assert.equal(await w.instances.clerk().userCount(), 0); + await assert.rejects(w.instances.apply(mfaGroup, quiet), /still applied/, 'one run drives on one instance at a time'); + await applied.release(); + assert.throws(() => w.instances.keys(), /no instance is applied/); + }); + + it('keeps the record and the state in private files, writes the record once, and keeps the platform key nowhere', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, w.say); + const dir = instancesDir(w); + const { name } = w.clerk.live()[0]!; + const record = join(dir, `${name}.json`); + const written = { bytes: readFileSync(record, 'utf8'), inode: statSync(record).ino }; + for (const group of [mfaGroup, orgGroup, standardGroup]) await (await w.instances.apply(group, w.say)).release(); + await w.open().ensure({ willChange: true }, w.say); + assert.deepEqual({ bytes: readFileSync(record, 'utf8'), inode: statSync(record).ino }, written, 'the record is written once'); + assert.deepEqual(readdirSync(dir).sort(), [`${name}.json`, `${name}.state.json`], 'no staging file is left behind'); + for (const file of readdirSync(dir).map((each) => join(dir, each))) assert.equal(statSync(file).mode & 0o777, 0o600, file); + assert.equal(statSync(dir).mode & 0o777, 0o700); + assert.equal(w.lines.join('\n').includes(PLATFORM_KEY), false); + assert.deepEqual(filesHolding(w, PLATFORM_KEY), []); + assert.equal(redact(`leaked ${PLATFORM_KEY}`), 'leaked ', 'a used platform key is redacted from any output line'); + }); + + it('keeps the secret key of its application in no file of the worktree or the home, from the create to the down', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, w.say); + const { name, sk, pk } = w.clerk.live()[0]!; + assert.deepEqual(filesHolding(w, sk), [], 'after the application is created'); + for (const group of [mfaGroup, standardGroup]) { + const applied = await w.instances.apply(group, w.say); + assert.equal(await w.instances.clerk().userCount(), 0, 'the run used the key'); + await applied.release(); + } + assert.deepEqual(filesHolding(w, sk), [], 'after a run in the command that created it'); + const run = w.open(); + await run.ensure(NOTHING_NEW, w.say); + const applied = await run.apply(mfaGroup, w.say); + assert.equal(await run.clerk().userCount(), 0, 'a run in a later command used the key it read from Clerk'); + await applied.release(); + assert.deepEqual(filesHolding(w, sk), [], 'after a run in a later command'); + const later = w.open(); + await later.ensure({ willChange: true }, w.say); + await later.doctorChecks({ live: false }, w.say); + assert.deepEqual(filesHolding(w, sk), [], 'after another up and doctor'); + assert.equal(w.lines.join('\n').includes(sk), false); + const record = JSON.parse(readFileSync(join(instancesDir(w), `${name}.json`), 'utf8')) as Record; + assert.deepEqual(record, { application: w.clerk.live()[0]!.id, instanceId: w.clerk.live()[0]!.instanceId, publishableKey: pk }); + await w.open().finish(w.workspace, { keepApplications: false }, quiet); + assert.deepEqual(filesHolding(w, sk), [], 'after down'); + }); + + it('reads the secret key from the Platform API into a value that output redacts', async () => { + const w = world(); + const unseen = 'sk_test_readFromClerkAndHeldInMemoryOnly'; + const application = heldWithoutState(w, 'aaaaaaaa', unseen); + assert.equal(redact(`Bearer ${unseen}`), `Bearer ${unseen}`, 'nothing has read it yet'); + const up = await w.open().ensure(NOTHING_NEW, quiet); + assert.deepEqual(up.map((view) => [idOf(view), view.created]), [[application.id, false]], 'the Backend API took the key that was read'); + assert.equal(redact(`Bearer ${unseen}`), 'Bearer '); + assert.deepEqual(filesHolding(w, unseen), []); + }); + + it('reads the secret key once in a command, however many calls use it, and again in the next command', async () => { + const w = world(); + const application = heldWithoutState(w, 'aaaaaaaa'); + const command = w.open(); + await command.ensure({ willChange: true }, quiet); + for (const group of [standardGroup, mfaGroup, orgGroup]) { + const applied = await command.apply(group, quiet); + await command.clerk().userCount(); + await command.clerk().usersOfRun(newRunId()); + await applied.release(); + } + await command.doctorChecks({ live: false }, quiet); + assert.deepEqual(secretKeyReads(w).map((request) => `${request.method} ${request.url.replace('api.clerk.com/v1/platform', '')}`), [readOf(application)]); + assert.ok(w.clerk.requests.filter((request) => request.url.startsWith('api.clerk.com/v1/users')).length >= 7, 'while the Backend API was called with it many times'); + await w.open().doctorChecks({ live: false }, quiet); + assert.equal(secretKeyReads(w).length, 2, 'the next command holds nothing from the last one'); + }); + + it('reads no secret key in the command that creates the application, because the create answer carries it', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const applied = await w.instances.apply(mfaGroup, quiet); + await w.instances.clerk().userCount(); + await applied.release(); + assert.deepEqual(secretKeyReads(w), []); + }); + + it('reads no secret key for a down, a dry look at what is held, or an application another run drives', async () => { + const w = world(); + const application = heldWithoutState(w, 'aaaaaaaa'); + assert.equal(w.open().recordedKey(), null); + const otherRun = 2000; + w.alive.add(otherRun); + writeFileSync(join(instancesDir(w), `${application.name}.state.json`), `${JSON.stringify({ environmentKey: STANDARD_ENVIRONMENT_KEY, settings: STANDARD, drivers: [{ pid: otherRun, startedAt: T0 }] })}\n`); + await w.open().ensure(NOTHING_NEW, quiet); + await assert.rejects(w.open().apply(standardGroup, quiet), { code: 'DEVICE_BUSY' }); + await w.open().finish(w.workspace, { keepApplications: true }, quiet); + await w.open().finish(w.workspace, { keepApplications: false }, quiet); + assert.deepEqual(secretKeyReads(w), []); + assert.equal(w.clerk.live().length, 0); + }); + + it('fails with the scope by name when the Platform API key may not read secret keys, and falls back to nothing on disk', async () => { + const w = world(); + const application = heldWithoutState(w, 'aaaaaaaa'); + w.clerk.state.scopes = PLATFORM_SCOPES.filter((scope) => scope !== SECRET_KEY_SCOPE); + const missingScope = (error: VerifyFailure): boolean => + error.code === 'KEYS_MISSING' && + error.message === `the Platform API key lacks the scope ${SECRET_KEY_SCOPE}, which reading the secret key of ${application.name} needs (403 authorization_missing_scopes)` && + error.fix === `add ${SECRET_KEY_SCOPE} to the Platform API key; the secret key of an application is kept on no disk, so each command that needs it reads it from Clerk`; + await assert.rejects(w.open().ensure(NOTHING_NEW, quiet), missingScope); + await assert.rejects(w.open().apply(standardGroup, quiet), missingScope); + const api = (await w.open().doctorChecks({ live: false }, quiet)).find((check) => check.id === 'clerk-api')!; + assert.deepEqual([api.ok, api.detail, api.fix], [false, `the Platform API key lacks the scope ${SECRET_KEY_SCOPE}, which reading the secret key of ${application.name} needs (403 authorization_missing_scopes)`, `add ${SECRET_KEY_SCOPE} to the Platform API key; the secret key of an application is kept on no disk, so each command that needs it reads it from Clerk`]); + assert.deepEqual(filesHolding(w, application.sk), [], 'the secret key is in no file'); + assert.deepEqual(w.clerk.requests.filter((request) => request.url.startsWith('api.clerk.com/v1/users')), [], 'no call went to the Backend API'); + assert.deepEqual(writes(w), [], 'the application was neither deleted nor replaced'); + assert.deepEqual(openApplications(w.workspace).map((entry) => entry.name), [application.name]); + w.clerk.state.scopes = PLATFORM_SCOPES; + assert.deepEqual((await w.open().ensure(NOTHING_NEW, quiet)).map((view) => [idOf(view), view.created]), [[application.id, false]], 'with the scope, the same application serves again'); + }); + + it('reports any other refusal of the secret key as the Platform API reports the rest', async () => { + const w = world(); + const application = heldWithoutState(w, 'aaaaaaaa'); + const answering = (status: number, body: unknown): typeof fetch => (async (input: string | URL, init?: RequestInit) => (String(input).endsWith('?include_secret_keys=true') ? new Response(JSON.stringify(body), { status }) : w.clerk.fetch(input, init))) as typeof fetch; + await assert.rejects( + w.open(1000, answering(500, { errors: [{ code: 'internal' }] })).ensure(NOTHING_NEW, quiet), + (error: VerifyFailure) => error.code === 'NOT_READY' && error.message === `Clerk's Platform API answered 500 internal to reading the secret key of ${application.name}` && error.fix === 'rerun; if it repeats, run `{cli} doctor`', + ); + await assert.rejects( + w.open(1000, answering(200, { application_id: application.id, instances: [{ environment_type: 'development', instance_id: application.instanceId, publishable_key: application.pk }] })).ensure(NOTHING_NEW, quiet), + (error: VerifyFailure) => error.code === 'NOT_READY' && error.message === `Clerk's answer about ${application.name} carries no secret key of a development instance`, + ); + assert.deepEqual(writes(w), []); + }); + + it('names the scopes Clerk reports as missing when it refuses the read, and the secret key scope when Clerk names none', async () => { + const w = world(); + const application = heldWithoutState(w, 'aaaaaaaa'); + const refusing = (meta: unknown): typeof fetch => + (async (input: string | URL, init?: RequestInit) => + String(input).endsWith('?include_secret_keys=true') ? new Response(JSON.stringify({ errors: [{ code: 'authorization_missing_scopes', ...(meta === undefined ? {} : { meta }) }] }), { status: 403 }) : w.clerk.fetch(input, init)) as typeof fetch; + const lacking = (named: string, added: string) => (error: VerifyFailure): boolean => + error.code === 'KEYS_MISSING' && + error.message === `the Platform API key lacks ${named}, which reading the secret key of ${application.name} needs (403 authorization_missing_scopes)` && + error.fix === `add ${added} to the Platform API key; the secret key of an application is kept on no disk, so each command that needs it reads it from Clerk`; + await assert.rejects(w.open(1000, refusing({ scopes: ['applications:read'] })).ensure(NOTHING_NEW, quiet), lacking('the scope applications:read', 'applications:read')); + await assert.rejects( + w.open(1000, refusing({ scopes: ['applications:read', SECRET_KEY_SCOPE] })).ensure(NOTHING_NEW, quiet), + lacking(`the scopes applications:read and ${SECRET_KEY_SCOPE}`, `applications:read and ${SECRET_KEY_SCOPE}`), + ); + for (const meta of [undefined, { scopes: [] }, { scopes: 'all' }]) await assert.rejects(w.open(1000, refusing(meta)).ensure(NOTHING_NEW, quiet), lacking(`the scope ${SECRET_KEY_SCOPE}`, SECRET_KEY_SCOPE)); + }); +}); + +describe('putting the application on the settings a group declares', () => { + it('drives three groups on one application with one create and three PATCHes in all', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, w.say); + const application = w.clerk.live()[0]!; + const seen: (readonly [string, boolean, unknown, unknown])[] = []; + for (const group of [standardGroup, mfaGroup, orgGroup]) { + const applied = await w.instances.apply(group, w.say); + assert.deepEqual(applied.instance, { id: application.id, name: application.name }); + assert.equal(applied.keys.pk, application.pk); + seen.push([group.settings.label, applied.changed, application.environment['user_settings.sign_up.mfa.required'], application.environment['organization_settings.force_organization_selection']]); + assert.equal(await applied.stillApplied(), true); + await applied.release(); + } + assert.deepEqual(seen, [['standard', false, false, false], [MFA_LABEL, true, true, false], [ORG_LABEL, true, false, true]], 'the next group returns what the last one changed to standard'); + assert.deepEqual(writes(w), ['POST /applications', `PATCH /applications/${application.id}/config`, `PATCH /applications/${application.id}/config`, `PATCH /applications/${application.id}/config`]); + assert.equal(w.clerk.applications.length, 1, 'never a second application'); + assert.deepEqual(patches(w)[1]!.body, MFA_BODY); + assert.deepEqual((patches(w)[2]!.body as { auth_multi_factor: unknown }).auth_multi_factor, standardFile().config.auth_multi_factor, 'the body is always the whole standard config with the declaration laid over it'); + assert.deepEqual(stateOf(w, application.name).settings, orgGroup.settings); + assert.ok(w.lines.includes(`settings standard already on ${application.id}`)); + assert.ok(w.lines.includes(`settings changing ${application.id} from standard to ${MFA_LABEL}, which ${MFA_SPEC} declares`)); + assert.ok(w.lines.includes(`settings ${MFA_LABEL} on ${application.id} in 0.0s (Clerk answered in 0.00s, the instance showed it 0.00s later), ${LEAVES} settings match`)); + assert.ok(w.lines.includes(`settings changing ${application.id} from ${MFA_LABEL} to ${ORG_LABEL}, which ${ORG_SPEC} declares`)); + }); + + it('returns to standard for the first spec of a standard group, and a later run finds the settings recorded', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const id = w.clerk.live()[0]!.id; + await (await w.instances.apply(mfaGroup, quiet)).release(); + const later = w.open(); + assert.equal(later.recordedKey(), mfaGroup.settings.key); + assert.deepEqual((await later.ensure(NOTHING_NEW, quiet)).map((view) => view.settings), [MFA_LABEL], 'up leaves the settings alone and says what they are'); + const before = patches(w).length; + await (await later.apply(mfaGroup, w.say)).release(); + assert.equal(patches(w).length, before, 'the same settings again cost no PATCH'); + await (await later.apply(standardGroup, w.say)).release(); + assert.ok(w.lines.includes(`settings changing ${id} from ${MFA_LABEL} to standard, for ${STANDARD_SPEC}`)); + assert.equal(w.clerk.live()[0]!.environment['user_settings.sign_up.mfa.required'], false); + }); + + it('sends the PATCH again after a crash between the PATCH and the record', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const application = w.clerk.live()[0]!; + let patched = false; + const dying = (async (input: string | URL, init?: RequestInit) => { + if (patched && String(input).includes('.clerk.accounts.dev')) throw new Error('the process died here'); + patched ||= init?.method === 'PATCH'; + return w.clerk.fetch(input, init); + }) as typeof fetch; + await assert.rejects(w.open(1000, dying).apply(mfaGroup, quiet), /the process died here/); + assert.equal(application.environment['user_settings.sign_up.mfa.required'], true, 'Clerk applied the change'); + assert.equal(stateOf(w, application.name).settings, undefined, 'and nothing records it'); + const before = patches(w).length; + await (await w.open().apply(mfaGroup, w.say)).release(); + assert.equal(patches(w).length, before + 1); + assert.ok(w.lines.includes(`settings changing ${application.id} from unknown settings to ${MFA_LABEL}, which ${MFA_SPEC} declares`)); + assert.equal(w.open().recordedKey(), mfaGroup.settings.key); + }); + + it('repairs recorded settings that the live environment contradicts', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const application = w.clerk.live()[0]!; + application.environment['user_settings.sign_up.mfa.required'] = true; + await w.open().ensure(NOTHING_NEW, w.say); + assert.ok(w.lines.includes(`settings changing ${application.id} from standard to standard, because it no longer shows standard`)); + assert.equal(application.environment['user_settings.sign_up.mfa.required'], false); + + await (await w.open().apply(mfaGroup, quiet)).release(); + application.environment['user_settings.sign_up.mfa.required'] = false; + const before = patches(w).length; + const applied = await w.open().apply(mfaGroup, quiet); + assert.equal(patches(w).length, before + 1, 'a record is trusted only after the live environment agrees'); + assert.notEqual(applied.changed, null); + assert.equal(application.environment['user_settings.sign_up.mfa.required'], true); + }); + + it('says after a group whether the settings still hold', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const applied = await w.instances.apply(mfaGroup, quiet); + assert.equal(await applied.stillApplied(), true); + const { environment } = w.clerk.live()[0]!; + environment['user_settings.sign_up.mfa.required'] = false; + assert.equal(await applied.stillApplied(), false); + environment['user_settings.sign_up.mfa.required'] = true; + assert.equal(await applied.stillApplied(), true); + environment['auth_config.test_mode'] = false; + assert.equal(await applied.stillApplied(), false, 'a required setting that no longer holds counts too'); + }); + + it('never retires an application over a torn or missing state file, and repairs it instead', async () => { + for (const damage of ['torn', 'missing'] as const) { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const application = w.clerk.live()[0]!; + const stateFile = join(instancesDir(w), `${application.name}.state.json`); + if (damage === 'torn') writeFileSync(stateFile, '{"settings":{"key":"'); + else rmSync(stateFile); + const keys = readFileSync(join(instancesDir(w), `${application.name}.json`), 'utf8'); + assert.equal(w.open().recordedKey(), null, damage); + const up = await w.open().ensure(NOTHING_NEW, quiet); + assert.deepEqual(up.map((view) => [idOf(view), view.created]), [[application.id, false]], damage); + assert.deepEqual(writes(w).slice(1), [`PATCH /applications/${application.id}/config`, `PATCH /applications/${application.id}/config`], `${damage}: one PATCH to create it and one to repair it, and no delete`); + assert.equal(readFileSync(join(instancesDir(w), `${application.name}.json`), 'utf8'), keys, damage); + assert.equal(w.open().recordedKey(), STANDARD.key, damage); + } + }); + + it('treats a record made against another standard file as no record', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const name = w.clerk.live()[0]!.name; + writeFileSync(join(instancesDir(w), `${name}.state.json`), JSON.stringify({ settings: { ...STANDARD, key: '000000000000' }, drivers: [] })); + assert.equal(w.open().recordedKey(), null); + }); + + it('returns an application recorded against another standard environment to the standard settings, and does not fail on what that file expected', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const application = w.clerk.live()[0]!; + const OTHER_RUN = 4242; + writeFileSync(join(instancesDir(w), `${application.name}.state.json`), JSON.stringify({ environmentKey: '000000000000', settings: STANDARD, drivers: [{ pid: OTHER_RUN, startedAt: T0 }] })); + assert.equal(w.open().recordedKey(), null, 'what the application is on is unknown'); + + w.alive.add(OTHER_RUN); + const before = patches(w).length; + await w.open().ensure(NOTHING_NEW, quiet); + assert.equal(patches(w).length, before, 'the run recorded as driving on it is still believed'); + + w.alive.clear(); + await w.open().ensure(NOTHING_NEW, w.say); + assert.ok(w.lines.includes(`settings changing ${application.id} from unknown settings to standard, because what it is on is not recorded`)); + assert.deepEqual(stateOf(w, application.name), { environmentKey: STANDARD_ENVIRONMENT_KEY, settings: STANDARD, drivers: [{ pid: OTHER_RUN, startedAt: T0 }] }); + }); + + it('moves an application whose state file is lost straight to the declared settings, with one request', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const application = w.clerk.live()[0]!; + rmSync(join(instancesDir(w), `${application.name}.state.json`)); + const before = patches(w).length; + await (await w.open().apply(mfaGroup, quiet)).release(); + assert.deepEqual(patches(w).slice(before).map((patch) => patch.body), [MFA_BODY], 'the body is the whole standard config with the declaration laid over it, so nothing has to come first'); + assert.deepEqual(stateOf(w, application.name).settings, mfaGroup.settings); + assert.equal(application.environment['user_settings.sign_up.mfa.required'], true); + }); + + it('drives and deletes an application the worktree holds with no state file, without creating one', async () => { + const w = world(); + const old = heldWithoutState(w, 'aaaaaaaa'); + const instances = w.open(); + const up = await instances.ensure(NOTHING_NEW, w.say); + assert.deepEqual(up.map((view) => [idOf(view), view.created, view.settings]), [[old.id, false, 'standard']]); + const applied = await instances.apply(mfaGroup, quiet); + assert.equal(applied.instance.id, old.id); + await applied.release(); + const finished = await w.open().finish(w.workspace, { keepApplications: false }, quiet); + assert.deepEqual(finished.map((application) => application.name), [old.name]); + assert.equal(w.clerk.live().length, 0); + assert.equal(writes(w).some((call) => call.startsWith('POST')), false, 'no create'); + assert.deepEqual(readdirSync(instancesDir(w)), []); + }); +}); + +describe('two runs in one worktree', () => { + const FIRST = 1000; + const SECOND = 2000; + + it('refuses a second run while the first still drives, on any settings, and makes no second application', async () => { + const w = world(); + const first = w.open(FIRST); + await first.ensure(NOTHING_NEW, quiet); + const held = await first.apply(mfaGroup, quiet); + w.alive.add(FIRST); + const application = w.clerk.live()[0]!; + const second = w.open(SECOND); + await second.ensure({ willChange: true }, quiet); + const before = w.clerk.platformRequests().length; + for (const group of [orgGroup, mfaGroup, standardGroup]) { + await assert.rejects( + second.apply(group, quiet), + (error: VerifyFailure) => + error.code === 'DEVICE_BUSY' && + RETRYABLE.has(error.code) && + error.message === `another run in this worktree (pid ${FIRST}) is driving on ${application.id}, and a worktree has one application, which serves one run at a time` && + error.fix === 'let that run finish, then rerun', + group.settings.label, + ); + } + assert.equal(w.clerk.platformRequests().length, before, 'it asked the Platform API for nothing: no second application, no change of settings'); + assert.equal(w.clerk.applications.length, 1); + assert.deepEqual(stateOf(w, application.name).drivers.map((driver) => driver.pid), [FIRST]); + assert.equal(await held.stillApplied(), true, 'the first run still has its settings'); + assert.throws(() => second.keys(), /no instance is applied/); + + await held.release(); + await assert.rejects(second.apply(orgGroup, quiet), { code: 'DEVICE_BUSY' }, 'between two groups of the first run the application is still its own'); + await first.stopDriving(); + const own = await second.apply(orgGroup, quiet); + assert.deepEqual(own.instance, { id: application.id, name: application.name }, 'once the first run has ended, the second drives on the same application'); + }); + + it('takes over the application of a run that died', async () => { + const w = world(); + const first = w.open(FIRST); + await first.ensure(NOTHING_NEW, quiet); + await first.apply(mfaGroup, quiet); + const application = w.clerk.live()[0]!; + assert.deepEqual(stateOf(w, application.name).drivers, [{ pid: FIRST, startedAt: T0 }]); + const own = await w.open(SECOND).apply(orgGroup, quiet); + assert.deepEqual(own.instance, { id: application.id, name: application.name }); + assert.equal(w.clerk.applications.length, 1); + assert.deepEqual(stateOf(w, application.name).drivers.map((driver) => driver.pid), [SECOND], 'a dead driver counts as nobody'); + }); + + it('keeps driving on the application it checked when it started, though its deadline comes within the hour meanwhile', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + w.clock.at = T0 + 6 * HOUR - 20 * 60_000; + for (const group of [mfaGroup, mfaGroup, orgGroup]) await (await w.instances.apply(group, quiet)).release(); + assert.equal(w.clerk.applications.length, 1, 'a run that started with more than an hour left ends before a reap can come'); + }); + + it('waits for the lock another command in the worktree holds before it ensures, applies, or stops driving', async () => { + const w = world(); + mkdirSync(join(w.workspace.root, 'locks'), { recursive: true }); + const whileLocked = async (what: string, start: () => Promise, untouched: () => void): Promise => { + const unlock = await takeSlotLock(join(w.workspace.root, 'locks', 'instances'), 0, () => new VerifyFailure('NOT_READY', 'the lock is taken', '')); + w.lines.length = 0; + let settled = false; + const pending = start().finally(() => (settled = true)); + while (!settled && !w.lines.includes(`wait another {cli} in this worktree (pid ${process.pid}) is creating, changing, or deleting instances; waiting for it, with no time limit`)) await new Promise((resolve) => setTimeout(resolve, 5)); + assert.equal(settled, false, `${what} went ahead while the lock was held`); + untouched(); + unlock(); + return pending; + }; + + await whileLocked('ensure', () => w.instances.ensure(NOTHING_NEW, quiet), () => assert.deepEqual(w.clerk.requests, [])); + const application = w.clerk.live()[0]!; + const applied = await whileLocked('apply', () => w.instances.apply(mfaGroup, quiet), () => assert.equal(patches(w).length, 1, 'only the PATCH that put it on the standard file')); + assert.equal(application.environment['user_settings.sign_up.mfa.required'], true); + await applied.release(); + await whileLocked('stopDriving', () => w.instances.stopDriving(), () => assert.deepEqual(stateOf(w, application.name).drivers.map((driver) => driver.pid), [1000])); + assert.deepEqual(stateOf(w, application.name).drivers, []); + }); + + it('leaves an application another run is driving completely alone before a run', async () => { + const w = world(); + const first = w.open(FIRST); + await first.ensure(NOTHING_NEW, quiet); + await first.apply(mfaGroup, quiet); + w.alive.add(FIRST); + const application = w.clerk.live()[0]!; + application.users = 99; + application.environment['user_settings.sign_up.mfa.required'] = false; + w.clock.at = T0 + 8 * HOUR; + const before = writes(w).length; + const up = await w.open(SECOND).ensure({ willChange: true }, w.say); + assert.deepEqual(writes(w).slice(before), [], 'not retired near its deadline or its user limit, not repaired, not reaped, not changed'); + assert.deepEqual(up.map((view) => [idOf(view), view.settings]), [[application.id, MFA_LABEL]]); + assert.equal(application.deleted, false); + }); +}); + +describe('a declaration Clerk or the instance does not take', () => { + const ready = async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + return { w, application: w.clerk.live()[0]! }; + }; + + it('names the key Clerk refused and the spec that declared it, and leaves the application as it was', async () => { + const { w, application } = await ready(); + w.clerk.state.refusals.push({ path: 'auth_multi_factor.required_for_sign_up', value: true, status: 400, code: 'unknown_config_key', param: 'auth_multi_factor.required_for_sign_up', message: 'is not a config key' }); + await assert.rejects( + w.instances.apply(mfaGroup, quiet), + (error: SettingsRefused) => + error instanceof SettingsRefused && + error.code === 'INSTANCE_MISCONFIGURED' && + error.message === `${MFA_SPEC} declares ${MFA_LABEL}, and Clerk's Platform API refused it: auth_multi_factor.required_for_sign_up (400 unknown_config_key): is not a config key` && + error.fix === `correct or remove \`auth_multi_factor.required_for_sign_up\` in \`config\` in ${MFA_FILE}: Clerk says what is wrong with it above.`, + ); + assert.deepEqual(writes(w).slice(-2), [`PATCH /applications/${application.id}/config`, `PATCH /applications/${application.id}/config?dry_run=true`], 'one dry run of the standard file alone, to see whose fault it is'); + assert.equal(w.instances.recordedKey(), STANDARD.key, 'a refused PATCH applied nothing, so the record stands'); + const before = patches(w).length; + await (await w.instances.apply(standardGroup, w.say)).release(); + assert.equal(patches(w).length, before, 'and the next group finds its settings without a PATCH'); + assert.throws(() => w.open().keys(), /no instance is applied/); + }); + + it('says to add what Clerk named when the declaration does not set it, and to correct it when the declaration sets it by its last segment', async () => { + const { w } = await ready(); + const consent: InstanceSettings = { config: { compliance: { legal_consent: { enabled: true } } }, environment: { 'user_settings.sign_up.legal_consent_enabled': true } }; + w.clerk.state.refusals.push({ path: 'compliance.legal_consent.enabled', value: true, status: 422, code: 'form_param_missing', param: ['terms_of_service_url', 'privacy_policy_url'], message: 'is required' }); + await assert.rejects( + w.instances.apply(groupOf(consent, MFA_SPEC), quiet), + (error: SettingsRefused) => + error instanceof SettingsRefused && + error.message === `${MFA_SPEC} declares compliance.legal_consent.enabled=true, and Clerk's Platform API refused it: terms_of_service_url (422 form_param_missing): is required` && + error.fix === `add \`terms_of_service_url\` to \`config\` in ${MFA_FILE} (Clerk requires it with what the spec declares), or remove what requires it.`, + ); + w.clerk.state.refusals.push({ path: 'auth_multi_factor.required_for_sign_up', value: true, status: 422, code: 'form_param_value_invalid', param: 'required_for_sign_up', message: 'is not allowed' }); + await assert.rejects(w.instances.apply(mfaGroup, quiet), (error: SettingsRefused) => error instanceof SettingsRefused && error.fix.startsWith(`correct or remove \`required_for_sign_up\` in \`config\` in ${MFA_FILE}: Clerk says what is wrong with it above.`)); + }); + + it('names every declared leaf when Clerk names no key', async () => { + const { w } = await ready(); + const both: InstanceSettings = { config: { auth_multi_factor: { required_for_sign_up: true }, auth_attack_protection: { pii_protection_enabled: false } }, environment: MFA.environment }; + w.clerk.state.refusals.push({ path: 'auth_attack_protection.pii_protection_enabled', value: false, status: 409, code: 'user_settings_invalid', message: 'the settings are not valid together' }); + await assert.rejects(w.instances.apply(groupOf(both, MFA_SPEC), quiet), (error: SettingsRefused) => error instanceof SettingsRefused && error.message.endsWith('refused it: 409 user_settings_invalid: the settings are not valid together') && error.fix === `Clerk refused these together; remove or correct one of auth_multi_factor.required_for_sign_up, auth_attack_protection.pii_protection_enabled in \`config\` in ${MFA_FILE}.`); + }); + + it('blames the standard file, not the spec, when Clerk refuses the standard file alone too', async () => { + const { w } = await ready(); + w.clerk.state.refusals.push({ path: 'auth_password.min_length', status: 400, code: 'config_key_body_invalid', param: 'auth_password.min_length', message: 'must be at least 10' }); + await assert.rejects( + w.instances.apply(mfaGroup, quiet), + (error: VerifyFailure) => + !(error instanceof SettingsRefused) && + error.code === 'INSTANCE_MISCONFIGURED' && + error.message === `Clerk's Platform API refused the standard settings in ${STANDARD_FILE}: auth_password.min_length (400 config_key_body_invalid): must be at least 10` && + error.fix === 'the standard file needs a change of its own; the spec is not at fault', + ); + const fresh = world(); + fresh.clerk.state.refusals.push({ path: 'auth_password.min_length', status: 400, code: 'config_key_body_invalid', param: 'auth_password.min_length', message: 'must be at least 10' }); + await assert.rejects(fresh.instances.ensure(NOTHING_NEW, quiet), (error: VerifyFailure) => !(error instanceof SettingsRefused) && error.message.startsWith(`Clerk's Platform API refused the standard settings in ${STANDARD_FILE}`)); + assert.equal(writes(fresh).some((call) => call.includes('dry_run')), false, 'the standard file was the body, so no second request is needed'); + }); + + it('fails a change that moved a required setting the declaration does not list as soon as two reads agree, and lists it ready to paste', async () => { + const { w, application } = await ready(); + w.clerk.state.alsoMoves = { 'auth_config.test_mode': false, 'user_settings.sign_up.captcha_enabled': true }; + w.slept.length = 0; + await assert.rejects( + w.instances.apply(mfaGroup, quiet), + (error: SettingsRefused) => + error instanceof SettingsRefused && + error.message === `${MFA_SPEC} declares ${MFA_LABEL}, and 0.5s after Clerk accepted it on ${application.id} the change moved 2 settings the declaration does not list: "auth_config.test_mode": false, "user_settings.sign_up.captcha_enabled": true` && + error.fix === `one setting can move several leaves: add them to \`environment\` in ${MFA_FILE}, as listed`, + ); + assert.deepEqual(w.slept, [500], 'the second read showed the same as the first, so it did not wait out the 15s'); + assert.equal(w.instances.recordedKey(), null, 'the application is on settings nobody declared, so the next use sends its own'); + w.clerk.state.alsoMoves = {}; + application.environment['auth_config.test_mode'] = true; + application.environment['user_settings.sign_up.captcha_enabled'] = false; + const both: InstanceSettings = { config: MFA.config, environment: { ...MFA.environment } }; + await (await w.instances.apply(groupOf(both, MFA_SPEC), quiet)).release(); + }); + + it('keeps waiting while a declared leaf does not show, and stops only when two reads in a row show the same rest', async () => { + const { w, application } = await ready(); + w.clerk.state.ignoreConfigKey = 'user_settings.sign_up.mfa.required'; + w.clerk.state.alsoMoves = { 'auth_config.test_mode': false, 'user_settings.sign_up.captcha_enabled': true }; + let reads = 0; + let patched = false; + const slow: typeof fetch = async (input, init) => { + patched ||= init?.method === 'PATCH'; + if (patched && String(input).endsWith('/v1/environment')) { + reads += 1; + if (reads === 3) application.environment['user_settings.sign_up.mfa.required'] = true; + if (reads === 4) application.environment['user_settings.sign_up.captcha_enabled'] = false; + } + return w.clerk.fetch(input, init); + }; + w.slept.length = 0; + await assert.rejects( + w.open(1000, slow).apply(mfaGroup, quiet), + (error: SettingsRefused) => error instanceof SettingsRefused && error.message === `${MFA_SPEC} declares ${MFA_LABEL}, and 2.0s after Clerk accepted it on ${application.id} the change moved 1 setting the declaration does not list: "auth_config.test_mode": false`, + ); + assert.equal(reads, 5, 'two reads without the declared leaf, one with it, one where the rest changed, and one that agreed with it'); + assert.deepEqual(w.slept, [500, 500, 500, 500]); + }); + + it('fails a change whose declared leaf never shows after the full wait, naming what the instance shows instead', async () => { + const { w, application } = await ready(); + w.clerk.state.ignoreConfigKey = 'user_settings.sign_up.mfa.required'; + w.slept.length = 0; + await assert.rejects( + w.instances.apply(mfaGroup, quiet), + (error: SettingsRefused) => error instanceof SettingsRefused && error.message === `${MFA_SPEC} declares ${MFA_LABEL}, and 15.0s after Clerk accepted it on ${application.id} it does not show user_settings.sign_up.mfa.required is false, and the declaration expects true` && error.fix.includes(`\`environment\` in ${MFA_FILE}`), + ); + assert.equal(w.slept.reduce((sum, ms) => sum + ms, 0), 15_000, 'identical reads that lack a declared leaf are not an answer yet'); + }); + + it('says that the instance has no such leaf when a declared leaf is misspelt, and gives a fix for it and for the leaf the change did move', async () => { + const { w, application } = await ready(); + const misspelt: InstanceSettings = { config: MFA.config, environment: { 'user_settings.sign_up.mfa.requird': true } }; + await assert.rejects( + w.instances.apply(groupOf(misspelt, MFA_SPEC), quiet), + (error: SettingsRefused) => + error instanceof SettingsRefused && + error.message === `${MFA_SPEC} declares ${MFA_LABEL}, and 15.0s after Clerk accepted it on ${application.id} its public environment has no leaf user_settings.sign_up.mfa.requird; and the change moved 1 setting the declaration does not list: "user_settings.sign_up.mfa.required": true` && + error.fix === `check the spelling of user_settings.sign_up.mfa.requird under \`environment\` in ${MFA_FILE}; one setting can move several leaves: add them to \`environment\` in ${MFA_FILE}, as listed`, + ); + }); + + it('fails when Clerk accepts a change and its answer does not hold it', async () => { + const { w } = await ready(); + w.clerk.state.dropFromAfter = 'auth_multi_factor.required_for_sign_up'; + await assert.rejects(w.instances.apply(mfaGroup, quiet), (error: VerifyFailure) => error.code === 'INSTANCE_MISCONFIGURED' && !(error instanceof SettingsRefused) && error.message.includes('its answer does not hold auth_multi_factor.required_for_sign_up=true (it has no such key)')); + }); + + it('blames the declaration when Clerk accepts a declared value and stores another, and the tool when it is a value no declaration set', async () => { + const { w } = await ready(); + const long: InstanceSettings = { config: { auth_password: { min_length: 200 } }, environment: { 'user_settings.password_settings.min_length': 200 } }; + w.clerk.state.storesInstead = { 'auth_password.min_length': 72 }; + await assert.rejects( + w.instances.apply(groupOf(long, MFA_SPEC), quiet), + (error: SettingsRefused) => + error instanceof SettingsRefused && + error.message === `${MFA_SPEC} declares auth_password.min_length=200, and Clerk stored 72` && + error.fix === `correct \`auth_password.min_length\` in \`config\` in ${MFA_FILE} to a value Clerk keeps`, + ); + assert.equal(w.instances.recordedKey(), null, 'the application is on a value nobody declared, so the next use sends its own settings'); + await assert.rejects(w.instances.apply(mfaGroup, quiet), (error: VerifyFailure) => error.code === 'INSTANCE_MISCONFIGURED' && !(error instanceof SettingsRefused) && error.message.includes('its answer does not hold auth_password.min_length=8 (it has 72)')); + }); + +}); + +describe('finishing a ledger', () => { + it('deletes every application, confirms each is absent from the list, and is safe to repeat', async () => { + const w = world(); + heldWithoutState(w, 'aaaaaaaa'); + const killed = heldWithoutState(w, 'bbbbbbbb'); + await w.open().ensure(NOTHING_NEW, quiet); + writeFileSync(join(instancesDir(w), `${killed.name}.state.json.0a1b2c3d.tmp`), '{}'); + const finished = await w.open().finish(w.workspace, { keepApplications: false }, w.say); + assert.equal(finished.length, 2); + assert.equal(w.clerk.live().length, 0); + assert.deepEqual(w.workspace.unclosedEntries(), []); + assert.deepEqual(readdirSync(instancesDir(w)), [], 'the record and state files die with the applications, and so does a state file a killed process left half written'); + const calls = platformCalls(w); + assert.equal(calls.filter((call) => call.startsWith('DELETE')).length, 2); + assert.equal(calls.at(-1), 'GET /applications', 'the list is read again after the deletes'); + const before = w.clerk.requests.length; + assert.deepEqual(await w.open().finish(w.workspace, { keepApplications: false }, quiet), []); + assert.equal(w.clerk.requests.length, before, 'a second finish finds nothing open and asks nothing'); + }); + + it('keeps the ledger entries when Clerk still lists the application', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + w.clerk.state.refuseDelete = true; + await assert.rejects(w.open().finish(w.workspace, { keepApplications: false }, quiet), (error: VerifyFailure) => /could not be deleted/.test(error.message) && error.fix.includes('down again')); + assert.equal(openApplications(w.workspace).length, 1); + w.clerk.state.refuseDelete = false; + await w.open().finish(w.workspace, { keepApplications: false }, quiet); + assert.equal(w.clerk.live().length, 0); + }); + + it('closes the identities that lived in an application with no Backend API call', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const run = newRunId(); + w.workspace.append({ id: newEntryId(), kind: 'user', run, userId: 'user_1', email: newTestEmail(run) }); + + const before = w.clerk.requests.length; + await w.open().finish(w.workspace, { keepApplications: false }, quiet); + assert.deepEqual(w.workspace.unclosedEntries(), [], 'the user the run ledgered is closed'); + assert.deepEqual(w.clerk.requests.slice(before).filter((request) => !request.url.startsWith('api.clerk.com/v1/platform/')), [], 'the users went with the application, so none was looked up or deleted'); + }); + + it('refuses an entry made in another workspace and leaves it open', async () => { + const w = world(); + const name = throwawayName(new Date(T0 + HOUR), 'aaaaaaaa'); + w.workspace.append({ id: newEntryId(), kind: 'application', name, workspace: 'org_elsewhere' }); + await assert.rejects( + w.instances.finish(w.workspace, { keepApplications: false }, quiet), + (error: VerifyFailure) => /belong to workspace org_elsewhere, which this credential does not reach/.test(error.message) && error.fix === `${name} belongs to workspace org_elsewhere, which this credential cannot reach; only a credential of that workspace can delete it`, + ); + assert.equal(openApplications(w.workspace).length, 1); + assert.ok(w.clerk.requests.every((request) => request.method === 'GET')); + }); + + it('removes the files of an application before it closes the entry, so the next up finishes a kill between the two', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const { name } = w.clerk.live()[0]!; + const append = w.workspace.append; + w.workspace.append = (entry) => { + if (entry.kind === 'done') throw new Error('the process died here'); + append(entry); + }; + await assert.rejects(w.open().finish(w.workspace, { keepApplications: false }, quiet), /the process died here/); + w.workspace.append = append; + assert.deepEqual(readdirSync(instancesDir(w)), [], 'the record is gone, though the entry is still open'); + assert.deepEqual(openApplications(w.workspace).map((entry) => entry.name), [name]); + await w.open().ensure(NOTHING_NEW, w.say); + assert.ok(w.lines.includes(`instance ${name} retired (its keys are lost)`)); + assert.equal(openApplications(w.workspace).some((entry) => entry.name === name), false); + }); + + it('still sends the delete when the list does not show an application it has the id for', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + let hidden = true; + const hiding = (async (input: string | URL, init?: RequestInit) => (hidden && (init?.method ?? 'GET') === 'GET' && String(input).endsWith('/platform/applications') ? new Response('[]', { status: 200 }) : w.clerk.fetch(input, init))) as typeof fetch; + await w.open(1000, hiding).finish(w.workspace, { keepApplications: false }, quiet); + hidden = false; + assert.equal(w.clerk.live().length, 0, 'a list that was wrongly empty did not leave a live application behind'); + }); + + it('finishes the ledger of a worktree that is gone', async () => { + const home = mkdtempSync(join(tmpdir(), 'verify-instances-home-')); + const gone = world({ home }); + await gone.instances.ensure(NOTHING_NEW, quiet); + rmSync(gone.dir, { recursive: true, force: true }); + const here = world({ home, shared: gone.clerk }); + await finishOrphanLedgers(home, here.dir, here.instances, here.say); + assert.equal(gone.clerk.live().length, 0); + assert.deepEqual(gone.workspace.unclosedEntries(), []); + assert.ok(here.lines.some((line) => /^reap ledger of .* 1 application/.test(line))); + assert.equal(here.lines.some((line) => line.startsWith('wait another')), false, 'another worktree\'s ledger waits for no lock here'); + }); + + it('leaves the applications alone when asked to keep them, and still closes their identities', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const run = newRunId(); + w.workspace.append({ id: newEntryId(), kind: 'user', run, userId: 'user_1', email: newTestEmail(run) }); + await w.open().finish(w.workspace, { keepApplications: true }, quiet); + assert.equal(w.clerk.live().length, 1); + assert.deepEqual(w.workspace.unclosedEntries().map((entry) => entry.kind), ['application']); + }); + +}); + +describe('reaping', () => { + it('deletes only applications past their own deadline, never this worktree\'s, and at most five a command', async () => { + const w = world(); + const expired = Array.from({ length: 7 }, (_, n) => w.clerk.plant(throwawayName(new Date(T0 - HOUR), `0000000${n}`))); + const fresh = w.clerk.plant(throwawayName(new Date(T0 + HOUR), 'ffffffff')); + const justPast = w.clerk.plant(throwawayName(new Date(T0 - 60_000), 'eeeeeeee')); + await w.instances.ensure(NOTHING_NEW, w.say); + assert.equal(expired.filter((a) => a.deleted).length, 5); + assert.equal(fresh.deleted, false, 'another session\'s live instance'); + assert.equal(justPast.deleted, false, 'inside the grace after its deadline'); + assert.equal(w.clerk.live().filter((a) => a.id.startsWith('app_fake')).length, 1, 'its own new instance'); + assert.ok(w.lines.some((line) => line.startsWith('reap 2 more expired applications'))); + }); + + it('does not reap the application it has just retired, or spend one of its five reaps on it', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const retired = w.clerk.live()[0]!; + const expired = Array.from({ length: 5 }, (_, n) => w.clerk.plant(throwawayName(new Date(T0 + HOUR), `0000000${n}`))); + w.clock.at = T0 + 7 * HOUR; + await w.open().ensure(NOTHING_NEW, w.say); + assert.ok(w.lines.includes(`instance ${retired.id} retired (its deadline is near)`)); + assert.equal(w.clerk.platformRequests().filter((request) => request.method === 'DELETE' && request.url.endsWith(`/${retired.id}`)).length, 1, 'one delete for the retired application'); + assert.deepEqual(expired.map((application) => application.deleted), [true, true, true, true, true], 'and all five reaps for the applications other sessions left'); + assert.deepEqual(w.lines.filter((line) => line.startsWith('reap ')).map((line) => line.split(/ +/)[1]), expired.map((application) => application.name)); + }); + + it('judges a deadline by Clerk\'s clock, not this machine\'s', async () => { + const dir = mkdtempSync(join(tmpdir(), 'verify-instances-')); + const clerk = fakeClerk({ now: () => T0 }); + const other = clerk.plant(throwawayName(new Date(T0 + HOUR), 'aaaaaaaa')); + const fastClock = T0 + 5 * HOUR; + const instances = createInstances({ workspace: openWorkspace({ packageDir: dir, worktree: dir, home: join(dir, 'home') }), env: { CLERK_PLATFORM_API_KEY: PLATFORM_KEY }, runner: noOp, progress: quiet, fetch: clerk.fetch, now: () => fastClock }); + await instances.ensure(NOTHING_NEW, quiet); + assert.equal(other.deleted, false); + }); + + it('reaps nothing when Clerk\'s answer carries no date', async () => { + const w = world(); + const expired = w.clerk.plant(throwawayName(new Date(T0 - 10 * HOUR), 'aaaaaaaa')); + w.clerk.state.noDate = true; + await w.instances.ensure(NOTHING_NEW, w.say); + assert.equal(expired.deleted, false); + }); + + it('does not fail the command when a reap is refused', async () => { + const w = world(); + w.clerk.plant(throwawayName(new Date(T0 - HOUR), 'bbbbbbbb')); + const refusing = (async (input: string | URL, init?: RequestInit) => (init?.method === 'DELETE' ? new Response(JSON.stringify({ errors: [{ code: 'internal' }] }), { status: 500 }) : w.clerk.fetch(input, init))) as typeof fetch; + await w.open(1000, refusing).ensure(NOTHING_NEW, w.say); + assert.ok(w.lines.some((line) => line.includes('left in place'))); + }); +}); + +describe('the Backend API', () => { + const keys = { pk: 'pk_test_x' as PublishableKey, sk: new Secret('clerk-secret-key', 'sk_test_ownInstanceKey123') }; + const answering = (status: number) => { + const calls: string[] = []; + const fetchImpl = (async (url: string | URL) => { + calls.push(new URL(String(url)).host); + return new Response(JSON.stringify(status === 200 ? [] : { errors: [{ code: status === 401 ? 'clerk_key_invalid' : 'internal' }] }), { status }); + }) as typeof fetch; + return { calls, backend: createClerkBackends(fetchImpl)(() => keys) }; + }; + const run = newRunId(); + + it('is api.clerk.com, called with the instance key', async () => { + const { calls, backend } = answering(200); + assert.deepEqual(await backend.usersOfRun(run), []); + assert.deepEqual(calls, ['api.clerk.com']); + }); + + it('fails a 401 with one message that names the likely cause and the fix, and asks no other host', async () => { + const { calls, backend } = answering(401); + await assert.rejects(backend.usersOfRun(run), keyRefused); + assert.deepEqual(calls, ['api.clerk.com']); + }); + + it('reports any other refusal as what Clerk answered', async () => { + await assert.rejects(answering(500).backend.usersOfRun(run), /^Error: Clerk GET \/users answered 500 \(internal\)$/); + }); + + it('brings no instance up in a sandbox whose credential replaces the instance key on api.clerk.com, says why in up and in doctor, and can still delete it', async () => { + const w = world({ env: {}, clerk: { attachesKey: true, replacesAuthorizationOnCom: true } }); + await assert.rejects(w.instances.ensure(NOTHING_NEW, w.say), keyRefused); + assert.ok(w.lines.some((line) => line.startsWith('instances a key attached outside this machine (no key is in this process) reaches'))); + const check = (await w.open().doctorChecks({ live: false }, quiet)).find((c) => c.id === 'clerk-api')!; + assert.deepEqual([check.ok, keyRefused(new VerifyFailure('NOT_READY', check.detail, check.fix ?? ''))], [false, true]); + assert.equal(w.clerk.requests.some((request) => request.url.startsWith('api.clerk.dev')), false); + assert.ok(w.clerk.platformRequests().every((request) => request.authorization === null)); + await w.open().finish(w.workspace, { keepApplications: false }, quiet); + assert.equal(w.clerk.live().length, 0); + }); +}); + +describe('doctor', () => { + const specFile = (w: World, path: string, source: string) => { + mkdirSync(join(w.dir, path, '..'), { recursive: true }); + writeFileSync(join(w.dir, path), source); + }; + const settingsCheck = async (w: World) => (await w.open().doctorChecks({ live: false }, quiet)).find((c) => c.id === 'settings')!; + + it('says which credential reaches which workspace and creates nothing', async () => { + const w = world(); + const checks = await w.instances.doctorChecks({ live: false }, quiet); + assert.deepEqual(checks.map((c) => [c.id, c.ok]), [['instances', true], ['clerk-api', true], ['settings', true]]); + assert.match(checks[0]!.detail, /^CLERK_PLATFORM_API_KEY reaches the verification workspace org_3KHungJxbvIscuSvy8oos5MHAli; none created yet$/); + assert.match(checks[1]!.detail, /^not observed yet/); + assert.equal(checks[2]!.detail, `none created yet; up creates one application from ${STANDARD_FILE}; 0 spec files declare settings`); + assert.deepEqual(platformCalls(w), ['GET /me', 'GET /applications']); + }); + + it('fails when the Platform API key lacks a scope, names every one it lacks, and asks Clerk nothing more', async () => { + const lacking = async (scopes: readonly string[] | undefined): Promise<{ readonly checks: readonly (readonly unknown[])[]; readonly calls: readonly string[]; readonly created: number }> => { + const w = world(); + w.clerk.state.scopes = scopes; + const checks = await w.instances.doctorChecks({ live: true }, quiet); + return { checks: checks.map((c) => [c.id, c.ok, c.detail, c.fix]), calls: platformCalls(w), created: w.clerk.applications.length }; + }; + assert.deepEqual(await lacking(PLATFORM_SCOPES.filter((scope) => scope !== SECRET_KEY_SCOPE)), { + checks: [['instances', false, `the Platform API key lacks the scope ${SECRET_KEY_SCOPE}, which verification needs`, `add ${SECRET_KEY_SCOPE} to the Platform API key`]], + calls: ['GET /me'], + created: 0, + }); + assert.deepEqual((await lacking(['applications:read', 'users:read'])).checks, [ + ['instances', false, `the Platform API key lacks the scopes applications:manage, applications:delete, and ${SECRET_KEY_SCOPE}, which verification needs`, `add applications:manage, applications:delete, and ${SECRET_KEY_SCOPE} to the Platform API key`], + ]); + assert.match((await lacking(undefined)).checks[0]![2] as string, /^the Platform API key lacks the scopes applications:read, applications:manage, applications:delete, and application_secret_keys:read,/, 'an answer that lists no scopes proves none'); + }); + + it('still reads a held application when the key lacks a scope, and says which', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + w.clerk.state.scopes = PLATFORM_SCOPES.filter((scope) => scope !== 'applications:delete'); + const checks = await w.open().doctorChecks({ live: false }, quiet); + assert.deepEqual(checks.map((c) => [c.id, c.ok]), [['instances', false], ['clerk-api', true], ['settings', true]]); + assert.match(checks[0]!.detail, /^app_fake1 \(verify-throwaway-until-\w+-[0-9a-f]{8}\) on standard; the Platform API key lacks the scope applications:delete, which verification needs$/); + assert.equal(checks[0]!.fix, 'add applications:delete to the Platform API key'); + }); + + it('names what a held application is on, which spec asked, and how many settings it compared', async () => { + const w = world(); + specFile(w, MFA_SPEC, "test('x', () => {});\n"); + specFile(w, MFA_FILE, JSON.stringify(MFA)); + specFile(w, STANDARD_SPEC, "test('x', () => {});\n"); + await w.instances.ensure(NOTHING_NEW, quiet); + const application = w.clerk.live()[0]!; + application.environment['auth_config.reverification'] = false; + const standard = await w.open().doctorChecks({ live: false }, quiet); + assert.match(standard[0]!.detail, new RegExp(`^CLERK_PLATFORM_API_KEY reaches the verification workspace ${WORKSPACE}; ${application.id} \\(${application.name}\\) on standard$`)); + assert.equal(standard[1]!.detail, `api.clerk.com accepts the own key of ${application.id}`); + assert.deepEqual([standard[2]!.id, standard[2]!.ok], ['settings', true]); + assert.equal(standard[2]!.detail, `${application.id} is on standard; ${LEAVES} settings match ${STANDARD_FILE}; 1 spec file declares settings`, 'a setting the file does not require is not reported'); + + await (await w.open().apply(mfaGroup, quiet)).release(); + const declared = await settingsCheck(w); + assert.equal(declared.ok, true); + assert.ok(declared.detail.startsWith(`${application.id} is on ${MFA_LABEL}, which ${MFA_SPEC} asked for; ${LEAVES} settings match ${STANDARD_FILE} with that declaration; `), declared.detail); + assert.ok((await w.open().doctorChecks({ live: false }, quiet))[0]!.detail.endsWith(` on ${MFA_LABEL}`)); + }); + + it('fails when a held application no longer shows its recorded settings, or has none recorded', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const application = w.clerk.live()[0]!; + application.environment['auth_config.test_mode'] = false; + const differing = await settingsCheck(w); + assert.equal(differing.ok, false); + assert.ok(differing.detail.startsWith(`${application.id} is recorded as on standard and shows auth_config.test_mode is false, and those settings expect true`), differing.detail); + assert.equal(differing.fix, '{cli} up returns it to the standard settings'); + application.environment['auth_config.test_mode'] = true; + rmSync(join(instancesDir(w), `${application.name}.state.json`)); + const unknown = await settingsCheck(w); + assert.deepEqual([unknown.ok, unknown.detail.split(';')[0]], [false, `${application.id} has no settings recorded`]); + application.deleted = true; + assert.match((await settingsCheck(w)).detail, new RegExp(`^Clerk no longer serves ${application.id}`)); + }); + + it('fails on the first spec file whose declaration a run would refuse, and names it', async () => { + const w = world(); + specFile(w, STANDARD_SPEC, "test('x', () => {});\n"); + specFile(w, 'specs/explored/a-stray.settings.json', JSON.stringify(MFA)); + specFile(w, 'specs/explored/c-malformed.e2e.ts', "test('x', () => {});\n"); + specFile(w, 'specs/explored/c-malformed.settings.json', "{ config: { auth_multi_factor: { required_for_sign_up: true } } }"); + specFile(w, 'specs/explored/d-unknown.e2e.ts', "test('x', () => {});\n"); + specFile(w, 'specs/explored/d-unknown.settings.json', JSON.stringify({ config: { auth_email: { nope: true } }, environment: MFA.environment })); + const stray = await settingsCheck(w); + assert.equal(stray.ok, false); + assert.match(stray.detail, /; specs\/explored\/a-stray\.settings\.json is not the settings file of a spec, so no run reads it$/); + rmSync(join(w.dir, 'specs/explored/a-stray.settings.json')); + const malformed = await settingsCheck(w); + assert.equal(malformed.ok, false); + assert.match(malformed.detail, /; specs\/explored\/c-malformed\.settings\.json: it is not JSON \(/); + assert.match(malformed.fix ?? '', /^write the settings of specs\/explored\/c-malformed\.e2e\.ts as one JSON object, as in `\{ "config": /); + rmSync(join(w.dir, 'specs/explored/c-malformed.settings.json')); + assert.match((await settingsCheck(w)).detail, /specs\/explored\/d-unknown\.e2e\.ts declares auth_email\.nope, and the standard file has no value to return it to$/); + }); + + it('says so when instances are held and the credential that down needs is gone', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const later = createInstances({ workspace: w.workspace, env: {}, runner: noOp, progress: quiet, fetch: w.clerk.fetch }); + const checks = await later.doctorChecks({ live: true }, quiet); + assert.equal(checks[0]!.ok, false); + assert.match(checks[0]!.detail, /^app_fake1 \(verify-throwaway-until-\w+-[0-9a-f]{8}\) on standard; no Clerk Platform API credential works here/); + assert.equal(checks.find((c) => c.id === 'settings')?.ok, true, 'the held application is still read'); + assert.equal(w.clerk.live().length, 1); + }); + + it('--live leaves an application this worktree already holds alone', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const checks = await w.open().doctorChecks({ live: true }, quiet); + const live = checks.find((c) => c.id === 'live-instance')!; + assert.deepEqual([live.ok, live.state], [true, 'not-run']); + assert.match(live.detail, /^not run: this worktree already holds app_fake1/); + assert.equal(w.clerk.live().length, 1); + assert.equal(w.clerk.applications.length, 1); + }); + + it('--live creates one application, checks it, deletes it, and leaves the ledger closed', async () => { + const w = world(); + const checks = await w.instances.doctorChecks({ live: true }, quiet); + assert.deepEqual(checks.map((c) => [c.id, c.ok]), [['instances', true], ['clerk-api', true], ['settings', true], ['live-instance', true]]); + assert.match(checks.find((c) => c.id === 'settings')!.detail, /^app_fake1 is on standard; \d+ settings match/); + assert.match(checks.find((c) => c.id === 'live-instance')!.detail, /^created app_fake1 \(verify-throwaway-until-\w+-[0-9a-f]{8}\), configured it, compared its environment, and deleted it \(1 application gone from the list\)/); + assert.equal(w.clerk.applications.length, 1); + assert.equal(w.clerk.live().length, 0); + assert.deepEqual(w.workspace.unclosedEntries(), []); + }); + + it('fails the instances check alone when no credential works, with one fix line that says how to supply a key and nothing else', async () => { + for (const live of [false, true]) { + const w = world({ env: {} }); + const checks = await w.instances.doctorChecks({ live }, quiet); + assert.deepEqual(checks.map((c) => [c.id, c.ok]), [['instances', false]]); + const fix = checks[0]!.fix ?? ''; + for (const way of ['set CLERK_PLATFORM_API_KEY to the team key', 'CLERK_PLATFORM_API_KEY_FILE', 'VERIFY_PLATFORM_KEY_REFERENCE']) assert.ok(fix.includes(way), way); + assert.deepEqual(fix.match(/op:\/\/[^;\s]*/g), [REFERENCE_SHAPE], 'the reference is shown as a shape, never with a vault or an item'); + let printed = ''; + createOutput(false, w.dir, 'bin/control-x', { write: (line: string) => (printed += line) }, { write: () => true }).result({ verb: 'doctor', ok: false, backend: { ios: 'local' }, checks }); + assert.equal(printed.split('\n').filter((line) => line.includes('fix:')).length, 1); + assert.equal(w.clerk.applications.length, 0); + } + }); +}); + +describe('the broker of a run', () => { + + it('creates a user only while an instance is applied, with that application\'s own key, and ledgers it', async () => { + const w = world(); + await w.instances.ensure(NOTHING_NEW, quiet); + const { run, scratch } = w.workspace.newRun(); + const broker = await startBroker(run, w.workspace, scratch as ScratchPath, { keys: () => w.instances.keys(), fetch: w.clerk.fetch }); + const create = async () => { + const response = await fetch(`${broker.url}/users`, { method: 'POST', headers: { Authorization: `Bearer ${readFileSync(broker.tokenFile, 'utf8')}` }, body: JSON.stringify({ email_address: [newTestEmail(run)], skip_password_requirement: true }) }); + return { status: response.status, json: (await response.json()) as Record }; + }; + try { + assert.equal((await create()).status, 502, 'nothing is applied before the run drives, so there is no key to forward with'); + const applied = await w.instances.apply(mfaGroup, quiet); + const created = await create(); + assert.equal(created.status, 200); + const ticket = await fetch(`${broker.url}/sign_in_tokens`, { method: 'POST', headers: { Authorization: `Bearer ${readFileSync(broker.tokenFile, 'utf8')}` }, body: JSON.stringify({ user_id: created.json.id, expires_in_seconds: 120 }) }); + assert.equal(ticket.status, 200); + await applied.release(); + } finally { + await broker.stop(); + } + const application = w.clerk.live()[0]!; + assert.equal(application.users, 1, 'the user was made with the applied application\'s own key'); + assert.deepEqual(w.workspace.unclosedEntries().flatMap((entry) => (entry.kind === 'user' ? [entry.run] : [])), [run], 'the create that was refused left no entry'); + }); + +}); + +describe('down with throwaway instances', () => { + function setup(platforms: readonly ('ios' | 'android')[]) { + const w = world(); + const backends = platforms.map((platform) => ({ kind: 'local', platform, availability: () => ({ usable: true, why: 'test' }), release: async () => undefined, reapable: async () => [], describe: (lease: LocalLease) => lease.deviceName }) as unknown as DeviceBackend); + const deps = (): Deps => ({ host: { ...host, platforms, backends } as unknown as HostAdapter, workspace: w.workspace, runner: async () => assert.fail('down runs no commands'), env: w.env, progress: w.say, instances: w.open() }); + for (const platform of platforms) w.workspace.writeLease({ backend: 'local', platform, slot: 1, deviceName: `verify-${platform}-1`, deviceId: `id-${platform}`, claimNonce: `claim-${platform}`, acquiredAt: '2026-10-05T12:00:00Z', installedBuild: null }); + return { w, deps }; + } + + it('deletes the applications and reports them', async () => { + const { w, deps } = setup(['ios']); + await w.instances.ensure(NOTHING_NEW, quiet); + const name = w.clerk.live()[0]!.name; + const before = w.clerk.requests.length; + const dry = await down(deps(), { verb: 'down', stale: false, dryRun: true }); + assert.ok(dry.dryRun); + assert.deepEqual(dry.wouldDelete, [{ kind: 'application', name }]); + assert.equal(w.clerk.requests.length, before, 'a dry run reads the ledger and asks Clerk nothing'); + const result = await down(deps(), { verb: 'down', stale: false, dryRun: false }); + assert.ok(!result.dryRun); + assert.deepEqual(result.deletedApplications, [{ name }]); + assert.equal(w.clerk.live().length, 0); + }); + + it('keeps them while another platform in the worktree still holds a lease', async () => { + const { w, deps } = setup(['ios', 'android']); + await w.instances.ensure(NOTHING_NEW, quiet); + const first = await down(deps(), { verb: 'down', platform: 'ios', stale: false, dryRun: false }); + assert.ok(!first.dryRun); + assert.deepEqual(first.deletedApplications, []); + assert.equal(w.clerk.live().length, 1); + assert.ok(w.lines.some((line) => line.includes('which its android lease still uses'))); + const second = await down(deps(), { verb: 'down', platform: 'android', stale: false, dryRun: false }); + assert.ok(!second.dryRun); + assert.equal(second.deletedApplications.length, 1); + assert.equal(w.clerk.live().length, 0); + }); + + it('deletes the applications even when the device would not release', async () => { + const { w, deps } = setup(['ios']); + await w.instances.ensure(NOTHING_NEW, quiet); + const stuck = deps(); + const failing = { ...stuck, host: { ...stuck.host, backends: stuck.host.backends.map((backend) => ({ ...backend, release: async () => assert.fail('the simulator would not shut down') })) } as unknown as HostAdapter }; + await assert.rejects(down(failing, { verb: 'down', stale: false, dryRun: false }), /would not shut down/); + assert.equal(w.clerk.live().length, 0, 'a lease that failed to release is not a reason to keep its instances'); + }); + + it('still releases the device and reports the failure when the applications cannot be deleted', async () => { + const { w, deps } = setup(['ios']); + await w.instances.ensure(NOTHING_NEW, quiet); + w.clerk.state.refuseDelete = true; + await assert.rejects(down(deps(), { verb: 'down', stale: false, dryRun: false }), /could not be deleted/); + assert.equal(w.workspace.readLease('ios'), null, 'the lease was released first'); + assert.equal(existsSync(instancesDir(w)), true); + rmSync(w.dir, { recursive: true, force: true }); + }); +}); + +describe('the key and child processes', () => { + const credentialVariables = { CLERK_PLATFORM_API_KEY: PLATFORM_KEY, CLERK_PLATFORM_API_KEY_FILE: '/home/x/key', VERIFY_PLATFORM_KEY_REFERENCE: REFERENCE }; + + it('takes the Platform API key and where it is kept out of the environment every child inherits, and keeps them for the instances layer', () => { + const env: NodeJS.ProcessEnv = { ...credentialVariables, PATH: '/usr/bin' }; + const kept = takePlatformKey(env); + assert.deepEqual(env, { PATH: '/usr/bin' }); + assert.deepEqual(kept, { ...credentialVariables, PATH: '/usr/bin' }); + }); + + it('gives a program it starts neither the platform key nor where it is kept', () => { + assert.deepEqual(withoutClerkKeys({ ...credentialVariables, HOME: '/home/x' }), { HOME: '/home/x' }); + }); + + it('redacts a key from the moment it is read, before anything sends it', () => { + new Secret('clerk-platform-key', 'ak_readButNeverUsed000000000000'); + assert.equal(redact('op said ak_readButNeverUsed000000000000'), 'op said '); + }); +}); diff --git a/integration/expo-native/test/lease-flow.test.ts b/integration/expo-native/test/lease-flow.test.ts new file mode 100644 index 00000000000..582f52e2908 --- /dev/null +++ b/integration/expo-native/test/lease-flow.test.ts @@ -0,0 +1,207 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { execFileSync } from 'node:child_process'; +import { existsSync, mkdirSync, mkdtempSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { HELD, heldInstances } from '../testing/fake-instances.ts'; +import { describe, it } from 'node:test'; +import { doctor, down, leaseForRun, up, type Deps } from '../src/core/verbs.ts'; +import { openWorkspace } from '../src/core/workspace.ts'; +import { VerifyFailure, type BuildKey, type Command, type DeviceBackend, type HostAdapter, type LocalLease, type ScratchPath } from '../src/core/types.ts'; + +function setup(buildMs: number, runtime?: HostAdapter['runtime']) { + const dir = mkdtempSync(join(tmpdir(), 'verify-flow-')); + execFileSync('git', ['init', '-q'], { cwd: dir }); + writeFileSync(join(dir, 'app.swift'), 'app'); + const events: string[] = []; + const progress: string[] = []; + let leases = 0; + const backend = { + kind: 'local', + platform: 'ios', + availability: () => ({ usable: true, why: 'test' }), + reapable: async () => [], + check: async () => 'held', + async acquire(request: { waitSeconds: number }): Promise { + events.push(`acquire wait=${request.waitSeconds}`); + leases += 1; + return { backend: 'local', platform: 'ios', slot: leases, deviceName: `verify-ios-${leases}`, deviceId: `UDID-${leases}`, claimNonce: `c${leases}`, acquiredAt: '', installedBuild: null }; + }, + install: async (lease: LocalLease) => (events.push('install'), lease), + release: async () => void events.push('release'), + describe: (lease: LocalLease) => lease.deviceName, + doctorChecks: async () => ({ toolchain: [], device: [] }), + } as unknown as DeviceBackend; + const host = { + repo: 'clerk-ios', + platforms: ['ios'], + backends: [backend], + appId: () => 'com.clerk.E2EHost', + buildInputs: () => ['app.swift'], + ...(runtime === undefined ? {} : { runtime }), + async build(platform: 'ios', key: BuildKey, into: ScratchPath) { + events.push('build'); + await new Promise((resolve) => setTimeout(resolve, buildMs)); + mkdirSync(into, { recursive: true }); + writeFileSync(join(into, 'E2EHost.app'), ''); + return { platform, key, appId: 'com.clerk.E2EHost', path: join(into, 'E2EHost.app') as ScratchPath, source: 'local' }; + }, + } as unknown as HostAdapter; + const workspace = openWorkspace({ packageDir: dir, worktree: dir, home: join(dir, 'home') }); + const deps: Deps = { + host, + workspace, + runner: async () => ({ code: 0, stdout: '', stderr: '' }), + env: {}, + progress: (line: string) => void progress.push(line), + instances: heldInstances(), + }; + return { deps, events, progress }; +} + +const runCommand: Extract = { verb: 'run', selection: { all: true }, video: false, retries: 0, githubReport: false, waitSeconds: 0 }; + +async function until(holds: () => boolean, what: string): Promise { + const deadline = Date.now() + 15_000; + while (!holds()) { + if (Date.now() > deadline) throw new Error(`timed out waiting for ${what}`); + await new Promise((resolve) => setTimeout(resolve, 10)); + } +} + +describe('lease flow', () => { + it('builds before it claims a lane', async () => { + const { deps, events } = setup(0); + await up(deps, { verb: 'up', waitSeconds: 0 }); + assert.deepEqual(events, ['build', 'acquire wait=0', 'install']); + }); + + it('doctor creates no directory and no file, in the worktree or in the machine-wide state', async () => { + const { deps } = setup(0); + const report = await doctor(deps, { verb: 'doctor', live: false }); + assert.equal(report.checks.find((c) => c.id === 'build')!.ok, false); + assert.equal(existsSync(deps.workspace.root), false); + assert.equal(existsSync(deps.workspace.home), false); + }); + + it('doctor names the platform in the build fix when the host has more than one, and the backend when the command forced it', async () => { + const { deps } = setup(0); + const fix = async (host: { readonly platforms?: HostAdapter['platforms'] }, command: Partial>) => (await doctor({ ...deps, host: { ...deps.host, ...host } }, { verb: 'doctor', live: false, ...command })).checks.find((c) => c.id === 'build')!.fix; + assert.equal(await fix({}, {}), '{cli} up'); + assert.equal(await fix({ platforms: ['ios', 'android'] }, {}), '{cli} up --platform ios'); + assert.equal(await fix({ platforms: ['ios', 'android'] }, { backend: 'local' }), '{cli} up --platform ios --backend local'); + assert.equal(await fix({}, { backend: 'local' }), '{cli} up --backend local'); + }); + + it('asks the host for the build inputs of the backend that was chosen', async () => { + const { deps } = setup(0); + const asked: string[] = []; + const host = { ...deps.host, buildInputs: (platform: string, backend: string) => (asked.push(`${platform} ${backend}`), ['app.swift']) } as HostAdapter; + await doctor({ ...deps, host }, { verb: 'doctor', live: false }); + await up({ ...deps, host }, { verb: 'up', waitSeconds: 0 }); + assert.deepEqual([...new Set(asked)], ['ios local']); + }); + + it('doctor only warns about a gh that cannot attach, and says what attach then cannot do', async () => { + const { deps } = setup(0); + const gh = (await doctor(deps, { verb: 'doctor', live: false })).checks.find((c) => c.id === 'gh-attach')!; + assert.deepEqual(gh, { + id: 'gh-attach', + ok: true, + state: 'warning', + detail: "this gh has no `gh pr comment --attach`, so `{cli} attach` cannot post a run's video and screenshots from this machine", + fix: 'install a gh build whose `gh pr comment` has --attach', + }); + }); + + it('leases no device when no Platform API credential works', async () => { + const { deps, events } = setup(0); + const noCredential: Deps['instances'] = { ...deps.instances, access: async () => Promise.reject(new VerifyFailure('KEYS_MISSING', 'no Clerk Platform API credential works here', 'set one')) }; + await assert.rejects(up({ ...deps, instances: noCredential }, { verb: 'up', waitSeconds: 0 }), (error: VerifyFailure) => error.code === 'KEYS_MISSING'); + assert.deepEqual(events, [], 'nothing was built or leased'); + }); + + it('keeps the lease and says so when the instances fail after the device was leased', async () => { + const { deps, events } = setup(0); + const failing = { ...deps.instances, ensure: async () => Promise.reject(new VerifyFailure('INSTANCE_MISCONFIGURED', 'the instance does not match its file', 'correct the file')) }; + await assert.rejects(up({ ...deps, instances: failing }, { verb: 'up', waitSeconds: 0 }), (error: VerifyFailure) => error.code === 'INSTANCE_MISCONFIGURED' && error.fix === 'correct the file; the device stays leased until `{cli} down`'); + assert.deepEqual(events, ['build', 'acquire wait=0', 'install'], 'the lease finished and reached the lease file'); + assert.notEqual(deps.workspace.readLease('ios'), null); + }); + + it('reports the instances it brought up', async () => { + const { deps } = setup(0); + assert.deepEqual((await up(deps, { verb: 'up', waitSeconds: 0 })).instances, [HELD]); + }); + + it('lets run join an up that is still building instead of failing DEVICE_BUSY', async () => { + const { deps, events, progress } = setup(400); + const building = up(deps, { verb: 'up', waitSeconds: 0 }); + await until(() => events.includes('build'), 'up to start building'); + progress.length = 0; + const device = await leaseForRun(deps, 'ios', runCommand, { willChange: false }, async (outcome) => outcome.lease.backend === 'local' && outcome.lease.deviceName); + await building; + assert.equal(device, 'verify-ios-1'); + assert.equal(progress.filter((l) => l.startsWith('wait')).length, 1, 'one wait line while up holds the lock'); + assert.match(progress.find((l) => l.startsWith('wait')) ?? '', /is building ios-[0-9a-f]{12} or leasing the device/); + assert.deepEqual(events, ['build', 'acquire wait=0', 'install'], 'run reused the lease up made'); + }); + + it('holds the device lock for the run, so a second run reports DEVICE_BUSY', async () => { + const { deps } = setup(0); + await leaseForRun(deps, 'ios', runCommand, { willChange: false }, async () => { + await assert.rejects(leaseForRun(deps, 'ios', runCommand, { willChange: false }, async () => undefined), (error: VerifyFailure) => { + assert.equal(error.code, 'DEVICE_BUSY'); + assert.match(error.fix, /\{cli\} run --all --wait /, 'the fix names run, the verb that takes --wait'); + return true; + }); + }); + await leaseForRun(deps, 'ios', runCommand, { willChange: false }, async () => undefined); + }); + + it('passes run --wait to the lane claim', async () => { + const { deps, events } = setup(0); + await leaseForRun(deps, 'ios', { ...runCommand, waitSeconds: 300 }, { willChange: false }, async () => undefined); + assert.ok(events.includes('acquire wait=300')); + }); + + it('hands on the dev server a host runtime returns and ledgers its processes once', async () => { + const devServer = 'http://127.0.0.1:8082'; + const metro = { what: 'metro' as const, pid: 4242, startedAt: Date.parse('2026-10-03T00:00:00Z') }; + const { deps, progress } = setup(0, async (_lease, say) => (say('metro starting'), { devServer, processes: [metro] })); + await up(deps, { verb: 'up', waitSeconds: 0 }); + assert.ok(progress.includes('metro starting'), 'what a runtime reports goes through the CLI\'s own progress output'); + assert.equal(await leaseForRun(deps, 'ios', runCommand, { willChange: false }, async (outcome) => outcome.devServer), devServer); + assert.equal(existsSync(join(deps.workspace.root, 'context.json')), false, 'a lease writes no file for the tests to read'); + const metros = deps.workspace.unclosedEntries().filter((e) => e.kind === 'process' && e.what === 'metro'); + assert.equal(metros.length, 1, 'a reused Metro is ledgered once'); + }); + + it('names no dev server for hosts without a runtime', async () => { + const { deps } = setup(0); + assert.equal(await leaseForRun(deps, 'ios', runCommand, { willChange: false }, async (outcome) => outcome.devServer), null); + }); + + it('makes down wait for a run that holds the device, with one wait line, instead of failing DEVICE_BUSY', async () => { + const { deps, progress } = setup(0); + let downFinished = false; + let releaseRun: () => void = () => undefined; + let driving = false; + const running = leaseForRun(deps, 'ios', runCommand, { willChange: false }, () => new Promise((resolve) => ((driving = true), (releaseRun = resolve)))); + await until(() => driving, 'the run to hold the device'); + progress.length = 0; + const downing = down(deps, { verb: 'down', stale: false, dryRun: false }).then((result) => { + downFinished = true; + return result; + }); + await until(() => progress.some((line) => line.startsWith('wait')), 'down to say it is waiting'); + assert.equal(downFinished, false, 'down is still waiting while the run holds the device'); + releaseRun(); + await running; + const result = await downing; + assert.equal(result.dryRun, false); + assert.equal(progress.filter((l) => l.startsWith('wait')).length, 1); + assert.match(progress.find((l) => l.startsWith('wait')) ?? '', /driving the device; down waits for it, with no time limit/); + }); +}); diff --git a/integration/expo-native/test/ledgers.test.ts b/integration/expo-native/test/ledgers.test.ts new file mode 100644 index 00000000000..88ede5c2f97 --- /dev/null +++ b/integration/expo-native/test/ledgers.test.ts @@ -0,0 +1,115 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { execFileSync } from 'node:child_process'; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { heldInstances } from '../testing/fake-instances.ts'; +import { describe, it } from 'node:test'; +import { ensureLease } from '../src/core/devices.ts'; +import { newEntryId, openWorkspace } from '../src/core/workspace.ts'; +import type { BuildKey, DeviceBackend, HostAdapter, LedgerEntry, LocalLease, ScratchPath } from '../src/core/types.ts'; + +function worktree(root: string, name: string): string { + const dir = join(root, name); + mkdirSync(dir, { recursive: true }); + execFileSync('git', ['init', '-q'], { cwd: dir }); + writeFileSync(join(dir, 'app.swift'), name); + return dir; +} + +const application = (name: string): LedgerEntry => ({ id: newEntryId(), kind: 'application', name, workspace: 'org_test' }); + +function fakes(finished: string[]) { + const lease: LocalLease = { backend: 'local', platform: 'ios', slot: 1, deviceName: 'verify-ios-1', deviceId: 'UDID', claimNonce: 'c', acquiredAt: '', installedBuild: null }; + const backend = { + kind: 'local', + platform: 'ios', + availability: () => ({ usable: true, why: 'test' }), + reapable: async () => [], + check: async () => 'held', + acquire: async () => lease, + install: async (held: LocalLease) => held, + describe: () => 'verify-ios-1', + } as unknown as DeviceBackend; + const host = { + repo: 'clerk-ios', + platforms: ['ios'], + backends: [backend], + appId: () => 'com.clerk.E2EHost', + buildInputs: () => ['app.swift'], + async build(platform: 'ios', key: BuildKey, into: ScratchPath) { + mkdirSync(into, { recursive: true }); + writeFileSync(join(into, 'E2EHost.app'), ''); + return { platform, key, appId: 'com.clerk.E2EHost', path: join(into, 'E2EHost.app') as ScratchPath, source: 'local' }; + }, + } as unknown as HostAdapter; + const instances = heldInstances({ finish: async (ledger) => (finished.push(ledger.worktree), []) }); + return { host, instances }; +} + +describe('up finishes ledgers of deleted worktrees', () => { + it('finishes their instances, closes their entries, and leaves live worktrees alone', async () => { + const root = mkdtempSync(join(tmpdir(), 'verify-ledgers-')); + const home = join(root, 'home'); + const live = worktree(root, 'live'); + const gone = worktree(root, 'gone'); + const other = worktree(root, 'other'); + const goneLedger = openWorkspace({ packageDir: gone, worktree: gone, home }); + goneLedger.append({ id: newEntryId(), kind: 'lease-intent', platform: 'ios', backend: 'local', worktree: gone }); + goneLedger.append(application('verify-throwaway-gone')); + const otherLedger = openWorkspace({ packageDir: other, worktree: other, home }); + otherLedger.append(application('verify-throwaway-other')); + rmSync(gone, { recursive: true }); + + const finished: string[] = []; + const { host, instances } = fakes(finished); + const workspace = openWorkspace({ packageDir: live, worktree: live, home }); + const options = { waitSeconds: 0, progress: () => undefined, instances, retryWith: '{cli} up --wait ' }; + await workspace.withAcquireLock('ios', (lock) => ensureLease(lock, undefined, workspace, host, options)); + + assert.deepEqual(finished, [gone]); + assert.deepEqual(openWorkspace({ packageDir: gone, worktree: gone, home }).unclosedEntries(), []); + assert.equal(otherLedger.unclosedEntries().length, 1, 'a worktree that still exists keeps its instances'); + + await workspace.withAcquireLock('ios', (lock) => ensureLease(lock, undefined, workspace, host, options)); + assert.equal(finished.length, 1, 'a finished ledger is not finished twice'); + }); + + it('leaves the ledger open when its instances cannot be deleted, so the next up retries', async () => { + const root = mkdtempSync(join(tmpdir(), 'verify-ledgers-')); + const home = join(root, 'home'); + const live = worktree(root, 'live'); + const gone = worktree(root, 'gone'); + openWorkspace({ packageDir: gone, worktree: gone, home }).append(application('verify-throwaway-gone')); + rmSync(gone, { recursive: true }); + const { host } = fakes([]); + const failing = heldInstances({ finish: async () => assert.fail('the Platform API is down') }); + const lines: string[] = []; + const workspace = openWorkspace({ packageDir: live, worktree: live, home }); + await workspace.withAcquireLock('ios', (lock) => ensureLease(lock, undefined, workspace, host, { waitSeconds: 0, progress: (l) => lines.push(l), instances: failing, retryWith: '{cli} up --wait ' })); + assert.equal(openWorkspace({ packageDir: gone, worktree: gone, home }).unclosedEntries().length, 1); + assert.ok(lines.some((l) => l.includes('left open'))); + }); + + it('reaps a removed worktree whose package lived at another path, using the path its ledger recorded', async () => { + const root = mkdtempSync(join(tmpdir(), 'verify-ledgers-')); + const home = join(root, 'home'); + const live = worktree(root, 'live'); + const gone = worktree(root, 'gone'); + const packageDir = join(gone, 'tools', 'device-tests'); + const goneLedger = openWorkspace({ packageDir, worktree: gone, home }); + goneLedger.append(application('verify-throwaway-gone')); + const owner = readFileSync(goneLedger.ledgerFile.replace(/\.jsonl$/, '.owner'), 'utf8').trim().split('\n'); + assert.deepEqual(owner, [gone, packageDir], 'the ledger records its worktree and its package directory'); + rmSync(gone, { recursive: true }); + const finished: string[] = []; + const { host, instances } = fakes(finished); + const workspace = openWorkspace({ packageDir: live, worktree: live, home }); + await workspace.withAcquireLock('ios', (lock) => + ensureLease(lock, undefined, workspace, host, { waitSeconds: 0, progress: () => undefined, instances, retryWith: '{cli} up --wait ' }), + ); + assert.deepEqual(finished, [gone]); + assert.deepEqual(openWorkspace({ packageDir, worktree: gone, home }).unclosedEntries(), []); + }); +}); diff --git a/integration/expo-native/test/lock.test.ts b/integration/expo-native/test/lock.test.ts new file mode 100644 index 00000000000..2dfcbf65af3 --- /dev/null +++ b/integration/expo-native/test/lock.test.ts @@ -0,0 +1,51 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { spawn } from 'node:child_process'; +import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { VerifyFailure } from '../src/core/types.ts'; +import { takeSlotLock } from '../src/core/workspace.ts'; + +const holder = join(import.meta.dirname, '..', 'testing', 'lock-holder.ts'); + +function runHolder(packageDir: string, home: string, log: string, startAt: number, mode: 'hold' | 'crash'): Promise { + return new Promise((resolve) => { + const child = spawn(process.execPath, [holder, packageDir, home, log, String(startAt), mode], { stdio: 'ignore' }); + child.on('close', (code) => resolve(code)); + }); +} + +describe('acquire lock', () => { + it('takes a lock whose file was cut short, which no running command can have left', async () => { + const dir = join(mkdtempSync(join(tmpdir(), 'verify-lock-')), 'lock'); + mkdirSync(dir); + writeFileSync(join(dir, '000000000001'), '{"pid":123,"star'); + const busy = () => new VerifyFailure('DEVICE_BUSY', 'the lock is taken', ''); + const release = await takeSlotLock(dir, 0, busy); + await assert.rejects(takeSlotLock(dir, 0, busy), { code: 'DEVICE_BUSY' }); + release(); + (await takeSlotLock(dir, 0, busy))(); + }); + + it('lets exactly one of several processes break a stale lock at a time', async () => { + for (let round = 0; round < 4; round += 1) { + const packageDir = mkdtempSync(join(tmpdir(), 'verify-lock-')); + const home = join(packageDir, 'home'); + const log = join(packageDir, 'log'); + writeFileSync(log, ''); + await runHolder(packageDir, home, log, 0, 'crash'); + const startAt = Date.now() + 1500; + const codes = await Promise.all(Array.from({ length: 6 }, () => runHolder(packageDir, home, log, startAt, 'hold'))); + assert.deepEqual(codes, [0, 0, 0, 0, 0, 0]); + const lines = readFileSync(log, 'utf8').trim().split('\n'); + assert.equal(lines.length, 12, `round ${round}: every holder entered once`); + for (let i = 0; i < lines.length; i += 2) { + const [enter, pid] = lines[i]!.split(' '); + assert.equal(enter, 'enter', `round ${round}: two holders overlapped:\n${lines.join('\n')}`); + assert.equal(lines[i + 1], `exit ${pid}`, `round ${round}: two holders overlapped:\n${lines.join('\n')}`); + } + } + }); +}); diff --git a/integration/expo-native/test/processes.test.ts b/integration/expo-native/test/processes.test.ts new file mode 100644 index 00000000000..bd2a3cbec16 --- /dev/null +++ b/integration/expo-native/test/processes.test.ts @@ -0,0 +1,45 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { execFileSync, spawn, type ChildProcess } from 'node:child_process'; +import { mkdtempSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { isAlive } from '../src/core/exec.ts'; +import { openProcesses, stopProcesses } from '../src/core/ledgers.ts'; +import { newEntryId, openWorkspace } from '../src/core/workspace.ts'; + +const startedAt = (pid: number) => Date.parse(execFileSync('ps', ['-o', 'lstart=', '-p', String(pid)], { encoding: 'utf8' }).trim()); + +function sleeper(): ChildProcess { + return spawn('sleep', ['30'], { stdio: 'ignore' }); +} + +async function exited(child: ChildProcess, ms: number): Promise { + if (child.exitCode !== null || child.signalCode !== null) return true; + return new Promise((resolve) => { + const timer = setTimeout(() => resolve(false), ms); + child.on('exit', () => { + clearTimeout(timer); + resolve(true); + }); + }); +} + +describe('stopProcesses', () => { + it('stops any ledgered process that is still the same process, whatever its command', async () => { + const dir = mkdtempSync(join(tmpdir(), 'verify-procs-')); + const workspace = openWorkspace({ packageDir: dir, worktree: dir, home: join(dir, 'home') }); + const recorder = sleeper(); + const reused = sleeper(); + await new Promise((resolve) => setTimeout(resolve, 200)); + workspace.append({ id: newEntryId(), kind: 'process', what: 'recorder', pid: recorder.pid!, startedAt: new Date(startedAt(recorder.pid!)).toISOString() }); + workspace.append({ id: newEntryId(), kind: 'process', what: 'agent-device', pid: reused.pid!, startedAt: new Date(startedAt(reused.pid!) - 3_600_000).toISOString() }); + const stopped = stopProcesses(workspace, openProcesses(workspace)); + assert.deepEqual(stopped, [`recorder ${recorder.pid}`, `agent-device ${reused.pid} had already exited`]); + assert.equal(await exited(recorder, 2000), true, 'the ledgered recorder was signalled'); + assert.equal(isAlive(reused.pid!), true, 'a pid whose start time does not match is left alone'); + assert.deepEqual(workspace.unclosedEntries(), []); + reused.kill(); + }); +}); diff --git a/integration/expo-native/test/run-groups.test.ts b/integration/expo-native/test/run-groups.test.ts new file mode 100644 index 00000000000..fe6427b2b71 --- /dev/null +++ b/integration/expo-native/test/run-groups.test.ts @@ -0,0 +1,669 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { execFileSync } from 'node:child_process'; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { describe, it } from 'node:test'; +import { exitCodeFor } from '../src/core/cli.ts'; +import { assertPublishable, readRecord } from '../src/core/evidence.ts'; +import { createInstances } from '../src/core/instances/instances.ts'; +import { SettingsRefused, settingsFileOf } from '../src/core/instances/settings.ts'; +import { commentBody } from '../src/core/publish.ts'; +import { runVerb, type Deps } from '../src/core/verbs.ts'; +import { openWorkspace } from '../src/core/workspace.ts'; +import { VerifyFailure, type BuildKey, type Command, type DeviceBackend, type HostAdapter, type InstanceSettings, type LocalLease, type ScratchPath } from '../src/core/types.ts'; +import { PLATFORM_KEY, fakeClerk, type FakeClerkOptions } from '../testing/fake-clerk.ts'; +import { LEFT_OUT, linesLeftOut, watchDriver } from '../src/core/driver.ts'; +import { readAgent } from '../specs/support/agent.ts'; + +const MFA: InstanceSettings = { config: { auth_multi_factor: { required_for_sign_up: true } }, environment: { 'user_settings.sign_up.mfa.required': true } }; +const FORCED_ORG: InstanceSettings = { config: { organization_settings: { force_organization_selection: true } }, environment: { 'organization_settings.force_organization_selection': true } }; +const MFA_LABEL = 'auth_multi_factor.required_for_sign_up=true'; +const ORG_LABEL = 'organization_settings.force_organization_selection=true'; +const AUTH_START = 'specs/golden/auth-start/auth-start.e2e.ts'; +const CHOOSE_ORG = 'specs/golden/session-tasks/choose-organization.e2e.ts'; +const COMPLETE_MFA = 'specs/golden/session-tasks/complete-setup-mfa.e2e.ts'; +const SETUP_MFA = 'specs/golden/session-tasks/setup-mfa.e2e.ts'; +const SIGN_UP = 'specs/golden/sign-up/complete.e2e.ts'; +const GOLDEN: Readonly> = { [AUTH_START]: null, [CHOOSE_ORG]: FORCED_ORG, [COMPLETE_MFA]: MFA, [SETUP_MFA]: MFA, [SIGN_UP]: null }; + +const FAKE_E2E = `#!${process.execPath} +const fs = require('node:fs'); +const path = require('node:path'); +const args = process.argv.slice(2); +const output = args[args.indexOf('--output') + 1]; +const specs = args.slice(1, args.indexOf('--config')); +const plan = JSON.parse(fs.readFileSync('fake-e2e.json', 'utf8')); +fs.appendFileSync('invocations.jsonl', JSON.stringify({ specs, output, args }) + '\\n'); +(async () => { + const token = fs.readFileSync(process.env.CLERK_E2E_API_TOKEN_FILE, 'utf8'); + const emailOf = () => 'verify_' + process.env.CLERK_E2E_RUN_ID.toLowerCase().replace(/[^a-z0-9]/g, '_') + '_' + require('node:crypto').randomBytes(4).toString('hex') + '+clerk_test@example.com'; + const stateDir = process.env.AGENT_DEVICE_STATE_DIR; + const driver = plan.driver; + if (driver) { + const registration = path.join(stateDir, 'daemon.json'); + const bootFails = (why) => { + console.log('boot failed: ' + why); + process.exitCode = 3; + }; + if (driver.refusesALeftoverRegistration && fs.existsSync(registration)) return bootFails('Daemon replacement could not be confirmed.'); + if ((driver.daemonFailsToStartIn || []).includes(path.basename(output))) { + fs.mkdirSync(stateDir, { recursive: true }); + fs.writeFileSync(registration, JSON.stringify({ pid: 2147483645, version: '0.21.22' })); + fs.writeFileSync(path.join(stateDir, 'daemon.log'), 'AGENT_DEVICE_DAEMON_PORT=51000\\nDaemon error: the daemon started for ' + path.basename(output) + ' gave up\\n'); + return bootFails('Failed to start daemon'); + } + if (driver.rewrites) for (const file of driver.rewrites) fs.writeFileSync(path.join(stateDir, file), ''); + if (driver.writes) { + const ask = (route, body) => fetch(process.env.CLERK_E2E_API_URL + route, { method: 'POST', headers: { Authorization: 'Bearer ' + token }, body: JSON.stringify(body) }).then((answer) => answer.json()); + const seededPassword = require('node:crypto').randomBytes(12).toString('hex') + 'Aa1!'; + const user = await ask('/users', { email_address: [emailOf()], password: seededPassword, bypass_client_trust: true }); + const ticket = (await ask('/sign_in_tokens', { user_id: user.id, expires_in_seconds: 60 })).token; + const part = (claims) => Buffer.from(JSON.stringify(claims)).toString('base64url'); + const sessionToken = [part({ alg: 'RS256', typ: 'JWT' }), part({ sid: 'sess_fake', sub: user.id }), require('node:crypto').randomBytes(32).toString('base64url')].join('.'); + const secrets = { ...driver.known, standInToken: token, seededPassword, ticket, sessionToken, runPassword: 'Verify-' + process.env.CLERK_E2E_RUN_ID + '-Pw1!' }; + fs.writeFileSync('secrets-the-driver-saw.json', JSON.stringify(secrets)); + const named = (line) => Object.entries(secrets).reduce((text, [name, value]) => text.split('{' + name + '}').join(value), line); + const expanded = (line) => { + const spelled = /\\{(each|pieces) (\\w+)\\}/.exec(line); + if (spelled === null) return [named(line)]; + const value = secrets[spelled[2]]; + const shown = spelled[1] === 'each' ? [...value] : value.match(/.{1,16}/g); + return shown.map((some) => named(line.replace(spelled[0], some))); + }; + for (const [file, lines] of Object.entries(driver.writes)) { + const target = file.startsWith('RUN/') ? path.join(path.dirname(output), file.slice(4)) : path.join(stateDir, file); + fs.mkdirSync(path.dirname(target), { recursive: true }); + fs.appendFileSync(target, lines.flatMap(expanded).join('\\n') + '\\n'); + } + } + } + const results = []; + for (const [index, file] of specs.entries()) { + console.log('fake e2e runs ' + file); + if ((plan.silent || []).includes(file)) continue; + const password = require('node:crypto').randomBytes(12).toString('hex') + 'Aa1!'; + const seeded = await fetch(process.env.CLERK_E2E_API_URL + '/users', { method: 'POST', headers: { Authorization: 'Bearer ' + token }, body: JSON.stringify({ email_address: [emailOf()], ...(plan.password === true ? { password, bypass_client_trust: true } : { skip_password_requirement: true }) }) }); + const body = await seeded.json(); + if (plan.password) { + fs.writeFileSync('password-the-spec-got', password); + console.log('the spec typed ' + password); + fs.mkdirSync(output, { recursive: true }); + fs.writeFileSync(path.join(output, 'trace.txt'), 'fill ' + password); + body.message = 'the field shows ' + password; + } + const passed = seeded.ok && !plan.password; + const shot = 'ios/' + index + '/attempt-0/screenshots/001-home.png'; + fs.mkdirSync(path.dirname(path.join(output, 'artifacts', shot)), { recursive: true }); + fs.writeFileSync(path.join(output, 'artifacts', shot), path.basename(output)); + results.push({ + id: path.basename(output) + '-' + index + '-0000', kind: 'test', titlePath: ['a test of ' + path.basename(file)], file, platform: 'ios', tags: [], + status: passed ? 'passed' : 'failed', + attempts: [{ status: passed ? 'passed' : 'failed', durationMs: 1000, ...(passed ? {} : { error: { message: body.message } }), artifacts: [{ kind: 'screenshot', path: shot, producer: { kind: 'step', stepId: 's1' } }], steps: [{ id: 's1', api: 'app.screenshot', label: 'home' }] }], + }); + } + if (plan.edit && output.endsWith('/e2e')) fs.appendFileSync(plan.edit, '\\n// edited while the run was in progress\\n'); + if (!(plan.noReport || []).includes(path.basename(output))) { + fs.mkdirSync(output, { recursive: true }); + fs.writeFileSync(path.join(output, 'report.json'), JSON.stringify({ schemaVersion: 'report-1', run: { results } })); + } + process.exitCode = plan.exitCode || 0; +})(); +`; + +interface FakeE2EPlan { + readonly silent?: readonly string[]; + readonly noReport?: readonly string[]; + readonly edit?: string; + readonly exitCode?: number; + readonly password?: boolean; + readonly driver?: { + readonly refusesALeftoverRegistration?: boolean; + readonly daemonFailsToStartIn?: readonly string[]; + readonly rewrites?: readonly string[]; + readonly known?: Readonly>; + readonly writes?: Readonly>; + }; +} + +const SPEC_SOURCE = "import { test } from '../../fixtures.ts';\n\ntest('x', async ({ host }) => {\n await host.seedUser();\n});\n"; +const settingsText = (declared: InstanceSettings): string => `${JSON.stringify(declared, null, 2)}\n`; + +function world(options: { readonly specs?: Readonly>; readonly plan?: FakeE2EPlan; readonly env?: Record; readonly clerk?: FakeClerkOptions; readonly logs?: () => Promise } = {}) { + const dir = mkdtempSync(join(tmpdir(), 'verify-run-')); + execFileSync('git', ['init', '-q'], { cwd: dir }); + writeFileSync(join(dir, 'app.swift'), 'app'); + for (const [path, declared] of Object.entries(options.specs ?? GOLDEN)) { + mkdirSync(dirname(join(dir, path)), { recursive: true }); + writeFileSync(join(dir, path), SPEC_SOURCE); + if (declared !== null) writeFileSync(join(dir, settingsFileOf(path)), settingsText(declared)); + } + mkdirSync(join(dir, 'node_modules', '.bin'), { recursive: true }); + writeFileSync(join(dir, 'node_modules', '.bin', 'e2e'), FAKE_E2E); + chmodSync(join(dir, 'node_modules', '.bin', 'e2e'), 0o755); + writeFileSync(join(dir, 'fake-e2e.json'), JSON.stringify(options.plan ?? {})); + + const clerk = fakeClerk(options.clerk); + const frontendApi = { answers: null as number | null }; + const request = (async (input: string | URL, init?: RequestInit) => (frontendApi.answers !== null && String(input).endsWith('/v1/environment') ? new Response('', { status: frontendApi.answers }) : clerk.fetch(input, init))) as typeof fetch; + const lines: string[] = []; + const hooks: ((line: string) => void)[] = []; + const progress = (line: string) => { + lines.push(line); + for (const hook of hooks) hook(line); + }; + let leases = 0; + const backend = { + kind: 'local', + platform: 'ios', + availability: () => ({ usable: true, why: 'test' }), + reapable: async () => [], + check: async () => 'held', + async acquire(): Promise { + leases += 1; + lines.push('device leased'); + return { backend: 'local', platform: 'ios', slot: leases, deviceName: `verify-ios-${leases}`, deviceId: `UDID-${leases}`, claimNonce: `c${leases}`, acquiredAt: '', installedBuild: null }; + }, + install: async (lease: LocalLease) => lease, + release: async () => undefined, + logs: options.logs ?? (async () => ''), + describe: (lease: LocalLease) => lease.deviceName, + } as unknown as DeviceBackend; + const host = { + repo: 'clerk-ios', + platforms: ['ios'], + backends: [backend], + appId: () => 'com.clerk.E2EHost', + buildInputs: () => ['app.swift'], + async build(platform: 'ios', key: BuildKey, into: ScratchPath) { + mkdirSync(into, { recursive: true }); + writeFileSync(join(into, 'E2EHost.app'), ''); + return { platform, key, appId: 'com.clerk.E2EHost', path: join(into, 'E2EHost.app') as ScratchPath, source: 'local' }; + }, + } as unknown as HostAdapter; + const workspace = openWorkspace({ packageDir: dir, worktree: dir, home: join(dir, 'home') }); + const env = options.env ?? { CLERK_PLATFORM_API_KEY: PLATFORM_KEY }; + const deps = (instanceEnv: Record = env): Deps => ({ + host, + workspace, + runner: async () => ({ code: 0, stdout: '', stderr: '' }), + env: {}, + progress, + instances: createInstances({ workspace, env: instanceEnv, runner: async () => ({ code: 127, stdout: '', stderr: '' }), progress, fetch: request }), + fetch: request, + }); + const invocations = () => readFileSync(join(dir, 'invocations.jsonl'), 'utf8').trim().split('\n').map((line) => JSON.parse(line) as { specs: string[]; output: string; args: string[] }); + const lastRecord = () => readRecord(workspace.runDir(workspace.runs().at(-1)!)); + const writes = () => clerk.platformRequests().filter((request) => request.method !== 'GET').map((request) => `${request.method} ${request.url.replace('api.clerk.com/v1/platform', '').replace(/\/app_\w+\/instances\/ins_\w+/, '/{app}')}`); + return { dir, clerk, frontendApi, workspace, lines, hooks, deps, invocations, lastRecord, writes }; +} + +const RUN_ALL: Extract = { verb: 'run', selection: { all: true }, video: false, retries: 0, githubReport: false, waitSeconds: 0 }; +const statuses = (results: readonly { readonly spec: { readonly path: string }; readonly title: string; readonly status: string }[]) => results.map((result) => `${result.status} ${result.spec.path.split('/').at(-1)}${result.title === 'not run' ? ' (not run)' : ''}`); + +describe('a run whose spec files declare different settings', () => { + it('reports to GitHub only when the command asks for it', async () => { + const actions = process.env.GITHUB_ACTIONS; + delete process.env.GITHUB_ACTIONS; + try { + const w = world(); + await runVerb(w.deps(), RUN_ALL); + assert.deepEqual(w.lines.filter((line) => line.startsWith('github')), []); + await runVerb(w.deps(), { ...RUN_ALL, githubReport: true }); + assert.deepEqual(w.lines.filter((line) => line.startsWith('github')), ['github not posted: not running on GitHub Actions']); + } finally { + if (actions !== undefined) process.env.GITHUB_ACTIONS = actions; + } + }); + + it('drives each group on one application, in one e2e invocation each, and records what it did', async () => { + const w = world(); + const result = await runVerb(w.deps(), RUN_ALL); + const { record, dir } = result; + const application = w.clerk.live()[0]!; + + assert.ok(w.lines.includes(`settings 3 groups in this run: standard (2 spec files), ${ORG_LABEL} (1), ${MFA_LABEL} (2)`), w.lines.join('\n')); + assert.deepEqual(w.invocations().map((invocation) => [invocation.output.split('/').at(-1), invocation.specs]), [['e2e', [AUTH_START, SIGN_UP]], ['e2e-2', [CHOOSE_ORG]], ['e2e-3', [COMPLETE_MFA, SETUP_MFA]]]); + assert.ok(w.invocations().every((invocation) => invocation.args.includes('--pass-with-no-tests')), 'a group whose tests are all left out must not fail the run'); + assert.deepEqual(w.writes(), ['POST /applications', 'PATCH /applications/{app}/config', 'PATCH /applications/{app}/config', 'PATCH /applications/{app}/config'], 'one create, and one PATCH for the standard file and for each of the two changes'); + assert.equal(w.clerk.applications.length, 1); + + assert.deepEqual(record.settings, [ + { label: 'standard', askedBy: null, specs: [AUTH_START, SIGN_UP], application: application.id, changed: false, held: true, e2eReport: join(dir, 'e2e', 'report.json') }, + { label: ORG_LABEL, askedBy: CHOOSE_ORG, specs: [CHOOSE_ORG], application: application.id, changed: true, held: true, e2eReport: join(dir, 'e2e-2', 'report.json') }, + { label: MFA_LABEL, askedBy: COMPLETE_MFA, specs: [COMPLETE_MFA, SETUP_MFA], application: application.id, changed: true, held: true, e2eReport: join(dir, 'e2e-3', 'report.json') }, + ]); + assert.equal(record.e2eReport, join(dir, 'e2e', 'report.json'), 'the first group\'s, so a run with one group reads as it always did'); + assert.deepEqual(statuses(record.results), ['passed auth-start.e2e.ts', 'passed complete.e2e.ts', 'passed choose-organization.e2e.ts', 'passed complete-setup-mfa.e2e.ts', 'passed setup-mfa.e2e.ts']); + assert.equal(JSON.stringify(record.results).includes('sourceHash'), false); + + assert.equal(application.users, 5, 'every spec seeded its user in the one application, with that application\'s key'); + assert.equal(record.identities.length, 5); + assert.ok(record.identities.every((identity) => identity.userId?.startsWith(`user_${application.id}_`))); + assert.equal(w.workspace.unclosedEntries().filter((entry) => entry.kind === 'user' && entry.run === record.run).length, 5, 'each spec made one user, and the run ledgered it'); + assert.deepEqual(record.identities.map((identity) => identity.email.startsWith(`verify_${record.run.replace(/-/g, '_')}_`)), [true, true, true, true, true]); + + const log = readFileSync(join(dir, 'e2e.log'), 'utf8').split('\n'); + assert.deepEqual(log.filter((line) => line.startsWith('settings ')), [`settings ${ORG_LABEL}: 1 spec file`, `settings ${MFA_LABEL}: 2 spec files`]); + assert.ok(log.indexOf(`fake e2e runs ${SIGN_UP}`) < log.indexOf(`settings ${ORG_LABEL}: 1 spec file`) && log.indexOf(`settings ${ORG_LABEL}: 1 spec file`) < log.indexOf(`fake e2e runs ${CHOOSE_ORG}`), 'one log for the whole run, in the order the groups ran'); + assert.deepEqual(record.screenshots, [{ label: 'home', path: join(dir, 'screenshots', 'home.png') }], 'one screenshot per label across the groups'); + assert.equal(readFileSync(join(dir, 'screenshots', 'home.png'), 'utf8'), 'e2e-3', 'the last group to take a label keeps it'); + + assert.match(result.next, /^\{cli\} attach /); + assert.equal(exitCodeFor(result), 0); + const comment = commentBody(assertPublishable(record, [])); + assert.ok(comment.includes(`Instance settings \`${ORG_LABEL}\` (declared by \`${CHOOSE_ORG}\`): 1 of 1 passed.`)); + assert.ok(comment.includes(`Instance settings \`${MFA_LABEL}\` (declared by \`${COMPLETE_MFA}\`): 2 of 2 passed.`)); + assert.equal(comment.includes('Instance settings `standard`'), false); + + w.lines.length = 0; + await runVerb(w.deps(), RUN_ALL); + assert.ok(w.lines.includes(`settings 3 groups in this run: ${MFA_LABEL} (2 spec files), ${ORG_LABEL} (1), standard (2)`), 'the next run starts with the settings the last one left'); + assert.ok(w.lines.includes(`settings ${MFA_LABEL} already on ${application.id}`)); + assert.equal(w.writes().length, 6, 'two more changes, and no create'); + }); + + it('runs one group in the directory a run always used, with nothing about settings to say', async () => { + const w = world({ specs: { [AUTH_START]: null, [SIGN_UP]: null } }); + const { record, dir } = await runVerb(w.deps(), RUN_ALL); + assert.deepEqual(w.invocations().map((invocation) => invocation.output.split('/').at(-1)), ['e2e']); + assert.equal(w.lines.some((line) => /groups in this run/.test(line)), false); + assert.deepEqual(record.settings.map((group) => [group.label, group.changed, group.e2eReport]), [['standard', false, join(dir, 'e2e', 'report.json')]], 'the application was put on the standard file before the run drove, so the group changed nothing'); + assert.equal(commentBody(assertPublishable(record, [])).includes('Instance settings'), false); + }); + + it('opens the Platform credential before it leases a device when the run will change settings', async () => { + const w = world(); + await w.deps().instances.ensure({ willChange: false }, () => undefined); + await assert.rejects(runVerb(w.deps({}), RUN_ALL), (error: VerifyFailure) => error.code === 'KEYS_MISSING'); + assert.equal(w.workspace.readLease('ios'), null, 'a credential that is gone stops the run before a device is leased, not between two groups'); + assert.equal(w.lines.includes('device leased'), false); + }); + +}); + +describe('a run whose spec seeds a user with a password', () => { + it('keeps the password out of everything the run writes, and marks a file that e2e wrote it into', async () => { + const gotFile = (dir: string) => join(dir, 'password-the-spec-got'); + const w = world({ specs: { [AUTH_START]: null }, plan: { password: true, exitCode: 1 }, logs: async () => `the app logged ${readFileSync(gotFile(w.dir), 'utf8')}` }); + const { record, dir } = await runVerb(w.deps(), RUN_ALL); + const password = readFileSync(gotFile(w.dir), 'utf8'); + assert.match(password, /^(?=.*[a-z])(?=.*[A-Z])(?=.*\d)(?=.*[^A-Za-z0-9]).{12,}$/, 'the spec was handed a password to type'); + + assert.equal(record.results[0]!.error, 'the field shows '); + assert.ok(readFileSync(join(dir, 'e2e.log'), 'utf8').includes('the spec typed ')); + assert.equal(readFileSync(record.appLog!, 'utf8'), 'the app logged '); + + const byE2E = [join(dir, 'e2e', 'report.json'), join(dir, 'e2e', 'trace.txt')]; + assert.deepEqual(record.tainted, byE2E, 'the two files e2e itself wrote the password into'); + assert.throws(() => assertPublishable(record, []), (error: VerifyFailure) => error.code === 'EVIDENCE_UNSAFE' && /has secret values in/.test(error.message)); + + const written = (at: string): string[] => readdirSync(at, { withFileTypes: true }).flatMap((entry) => (entry.isDirectory() ? written(join(at, entry.name)) : entry.isFile() ? [join(at, entry.name)] : [])); + const holding = written(w.dir).filter((file) => readFileSync(file).includes(password)); + assert.deepEqual(holding.sort(), [...byE2E, gotFile(w.dir)].sort(), 'no ledger, record, log, or scratch file holds it'); + }); +}); + +describe('a group that did not run in full', () => { + const notPublishable = (w: ReturnType) => assert.throws(() => assertPublishable(w.lastRecord(), []), { code: 'EVIDENCE_UNSAFE' }); + + it('fails every file of a group whose e2e wrote no report, seals the run, and exits non-zero', async () => { + const w = world({ plan: { noReport: ['e2e-2'] } }); + await assert.rejects(runVerb(w.deps(), RUN_ALL), (error: VerifyFailure) => error.code === 'E2E_CRASHED' && error.message === `e2e exited 0 before writing a report for ${ORG_LABEL}` && /e2e\.log$/.test(error.fix)); + const record = w.lastRecord(); + assert.equal(record.sealed, true); + assert.deepEqual(statuses(record.results), ['passed auth-start.e2e.ts', 'passed complete.e2e.ts', 'failed choose-organization.e2e.ts (not run)', 'passed complete-setup-mfa.e2e.ts', 'passed setup-mfa.e2e.ts']); + assert.equal(record.results[2]!.error, `e2e exited 0 before writing a report for ${ORG_LABEL}`); + assert.deepEqual(record.settings.map((group) => group.held), [true, true, true]); + notPublishable(w); + assert.ok(w.lines.some((line) => /^evidence .* sealed; 1 group of 3 did not run in full/.test(line))); + }); + + it('seals the run with every group\'s result when the device logs cannot be read, and says so in app.log', async () => { + const sessionEnded = async (): Promise => { + throw new Error('the session has ended'); + }; + const w = world({ plan: { noReport: ['e2e-2'] }, logs: sessionEnded }); + await assert.rejects(runVerb(w.deps(), RUN_ALL), (error: VerifyFailure) => error.code === 'E2E_CRASHED' && error.message === `e2e exited 0 before writing a report for ${ORG_LABEL}`); + const record = w.lastRecord(); + assert.equal(record.sealed, true); + assert.deepEqual(statuses(record.results), ['passed auth-start.e2e.ts', 'passed complete.e2e.ts', 'failed choose-organization.e2e.ts (not run)', 'passed complete-setup-mfa.e2e.ts', 'passed setup-mfa.e2e.ts']); + assert.equal(readFileSync(record.appLog!, 'utf8'), 'the device logs could not be read: the session has ended'); + + const passing = world({ specs: { [AUTH_START]: null }, logs: sessionEnded }); + const result = await runVerb(passing.deps(), RUN_ALL); + assert.equal(exitCodeFor(result), 0, 'a failed log read alone does not fail the run'); + assert.equal(readFileSync(result.record.appLog!, 'utf8'), 'the device logs could not be read: the session has ended'); + }); + + it('fails a selected file that has no result in its group\'s report', async () => { + const w = world({ plan: { silent: [SETUP_MFA] } }); + const result = await runVerb(w.deps(), RUN_ALL); + assert.deepEqual(statuses(result.record.results).slice(3), ['passed complete-setup-mfa.e2e.ts', 'failed setup-mfa.e2e.ts (not run)']); + assert.equal(result.record.results[4]!.error, 'e2e reported no result for this file: it registers no test'); + assert.equal(exitCodeFor(result), 1); + assert.doesNotMatch(result.next, /attach/); + notPublishable(w); + }); + + it('says that e2e exited non-zero when a selected file has no result, and where to read why', async () => { + const w = world({ plan: { silent: [SETUP_MFA], exitCode: 1 } }); + const result = await runVerb(w.deps(), RUN_ALL); + assert.equal(result.record.results[4]!.error, 'e2e exited 1 and reported no result for this file: it failed to load or registers no test; e2e.log in the run directory says which'); + }); + + it('fails the group of a spec file edited after the run was planned, and still runs the others', async () => { + const w = world(); + writeFileSync(join(w.dir, 'fake-e2e.json'), JSON.stringify({ edit: join(w.dir, SETUP_MFA) })); + await assert.rejects(runVerb(w.deps(), RUN_ALL), (error: VerifyFailure) => error.code === 'USAGE' && error.message === `${SETUP_MFA} or its settings file changed while the run was in progress` && error.fix.startsWith('rerun')); + const record = w.lastRecord(); + assert.deepEqual(statuses(record.results), ['passed auth-start.e2e.ts', 'passed complete.e2e.ts', 'passed choose-organization.e2e.ts', 'failed complete-setup-mfa.e2e.ts (not run)', 'failed setup-mfa.e2e.ts (not run)']); + assert.deepEqual(record.settings[2], { label: MFA_LABEL, askedBy: COMPLETE_MFA, specs: [COMPLETE_MFA, SETUP_MFA], application: null, changed: false, held: false, e2eReport: null }); + assert.equal(w.invocations().length, 2); + assert.equal(w.clerk.live()[0]!.environment['user_settings.sign_up.mfa.required'], false, 'the settings of a plan that no longer matches the file were never applied'); + }); + + it('fails the group of a spec whose settings file was edited while its settings were being applied, before e2e runs it, and still runs the next', async () => { + const w = world(); + w.hooks.push((line) => { + if (line.startsWith('settings changing ') && line.endsWith(`to ${ORG_LABEL}, which ${CHOOSE_ORG} declares`)) writeFileSync(join(w.dir, settingsFileOf(CHOOSE_ORG)), settingsText(MFA)); + }); + await assert.rejects(runVerb(w.deps(), RUN_ALL), (error: VerifyFailure) => error.code === 'USAGE' && error.message === `${CHOOSE_ORG} or its settings file changed while the run was in progress` && error.fix.startsWith('rerun')); + const record = w.lastRecord(); + assert.deepEqual(w.invocations().map((invocation) => invocation.specs), [[AUTH_START, SIGN_UP], [COMPLETE_MFA, SETUP_MFA]], 'e2e never ran the file under the settings of its old declaration'); + assert.deepEqual(statuses(record.results), ['passed auth-start.e2e.ts', 'passed complete.e2e.ts', 'failed choose-organization.e2e.ts (not run)', 'passed complete-setup-mfa.e2e.ts', 'passed setup-mfa.e2e.ts']); + assert.deepEqual(record.settings.map((group) => [group.application !== null, group.changed, group.held, group.e2eReport !== null]), [[true, false, true, true], [true, true, false, false], [true, true, true, true]]); + assert.deepEqual(w.workspace.unclosedEntries().filter((entry) => entry.kind === 'application').length, 1); + const state = JSON.parse(readFileSync(join(w.workspace.root, 'instances', `${w.clerk.live()[0]!.name}.state.json`), 'utf8')) as { drivers: unknown[] }; + assert.deepEqual(state.drivers, [], 'the run records itself as driving for all its groups and no longer once it has ended'); + }); + + it('fails the group of a spec file edited while e2e was running it, though e2e reported a pass, and still runs the next', async () => { + const w = world(); + writeFileSync(join(w.dir, 'fake-e2e.json'), JSON.stringify({ edit: join(w.dir, AUTH_START) })); + await assert.rejects(runVerb(w.deps(), RUN_ALL), (error: VerifyFailure) => error.code === 'USAGE' && error.message === `${AUTH_START} or its settings file changed while the run was in progress` && error.fix.startsWith('rerun')); + const record = w.lastRecord(); + assert.deepEqual(statuses(record.results), ['failed auth-start.e2e.ts (not run)', 'failed complete.e2e.ts (not run)', 'passed choose-organization.e2e.ts', 'passed complete-setup-mfa.e2e.ts', 'passed setup-mfa.e2e.ts']); + assert.deepEqual(record.settings.map((group) => [group.application !== null, group.held, group.e2eReport !== null]), [[true, true, true], [true, true, true], [true, true, true]]); + assert.equal(w.invocations().length, 3); + notPublishable(w); + }); + + it('keeps in the run directory the spec and the settings each group was planned from, not what the files hold later', async () => { + const w = world(); + w.hooks.push((line) => { + if (line.startsWith('instances ')) writeFileSync(join(w.dir, settingsFileOf(CHOOSE_ORG)), settingsText(MFA)); + }); + await assert.rejects(runVerb(w.deps(), RUN_ALL), (error: VerifyFailure) => error.message === `${CHOOSE_ORG} or its settings file changed while the run was in progress`); + const run = w.workspace.runDir(w.workspace.runs().at(-1)!); + assert.equal(readFileSync(join(run, settingsFileOf(CHOOSE_ORG)), 'utf8'), settingsText(FORCED_ORG)); + assert.equal(readFileSync(join(run, CHOOSE_ORG), 'utf8'), SPEC_SOURCE); + assert.equal(existsSync(join(run, settingsFileOf(AUTH_START))), false, 'a spec on the standard settings has no settings file to keep'); + }); + + it('refuses to run while a JSON file under specs is not the settings file of a spec, before it leases anything', async () => { + const w = world(); + writeFileSync(join(w.dir, 'specs/golden/session-tasks/setup-mfa.e2e.settings.json'), settingsText(MFA)); + await assert.rejects(runVerb(w.deps(), RUN_ALL), (error: VerifyFailure) => error.code === 'USAGE' && error.message === 'specs/golden/session-tasks/setup-mfa.e2e.settings.json is not the settings file of a spec, so no run reads it' && error.fix.includes('.settings.json in place of .e2e.ts')); + assert.equal(w.lines.includes('device leased'), false); + }); + + it('stops at a failure that comes after the settings were applied, and fails every later group with it', async () => { + const w = world(); + w.hooks.push((line) => { + if (line.startsWith('settings changing ') && line.includes(` to ${ORG_LABEL}, `)) rmSync(join(w.dir, 'node_modules', '.bin', 'e2e')); + }); + await assert.rejects(runVerb(w.deps(), RUN_ALL), (error: VerifyFailure) => error.code === 'NOT_READY' && error.message === 'the pinned e2e is not installed'); + const record = w.lastRecord(); + assert.deepEqual(statuses(record.results), ['passed auth-start.e2e.ts', 'passed complete.e2e.ts', 'failed choose-organization.e2e.ts (not run)', 'failed complete-setup-mfa.e2e.ts (not run)', 'failed setup-mfa.e2e.ts (not run)']); + assert.equal(record.results[4]!.error, 'an earlier group of this run failed, so this one did not run: the pinned e2e is not installed'); + assert.deepEqual(record.settings.map((group) => [group.application !== null, group.changed, group.held, group.e2eReport !== null]), [[true, false, true, true], [true, true, false, false], [false, false, false, false]]); + assert.equal(w.writes().filter((call) => call.startsWith('PATCH')).length, 2, 'the standard file and the one change; the last group\'s settings were never sent'); + }); + + it('fails only its own group when Clerk refuses the declaration, and goes on to the next', async () => { + const w = world(); + w.clerk.state.refusals.push({ path: 'organization_settings.force_organization_selection', value: true, status: 400, code: 'unknown_config_key', param: 'organization_settings.force_organization_selection', message: 'is not a config key' }); + await assert.rejects(runVerb(w.deps(), RUN_ALL), (error: VerifyFailure) => error instanceof SettingsRefused && error.message.startsWith(`${CHOOSE_ORG} declares ${ORG_LABEL}, and Clerk's Platform API refused it`)); + const record = w.lastRecord(); + assert.deepEqual(statuses(record.results), ['passed auth-start.e2e.ts', 'passed complete.e2e.ts', 'failed choose-organization.e2e.ts (not run)', 'passed complete-setup-mfa.e2e.ts', 'passed setup-mfa.e2e.ts']); + assert.match(record.results[2]!.error ?? '', /refused it: organization_settings\.force_organization_selection \(400 unknown_config_key\)/); + assert.deepEqual(record.settings.map((group) => [group.application !== null, group.changed, group.held, group.e2eReport !== null]), [[true, false, true, true], [false, false, false, false], [true, true, true, true]]); + assert.deepEqual(w.invocations().map((invocation) => invocation.specs), [[AUTH_START, SIGN_UP], [COMPLETE_MFA, SETUP_MFA]]); + notPublishable(w); + }); + + it('stops at a failure that is not about a declaration, and fails every later group with it', async () => { + const w = world(); + const deps = w.deps(); + await deps.instances.ensure({ willChange: false }, () => undefined); + w.clerk.state.failConfigure = 1; + await assert.rejects(runVerb(deps, RUN_ALL), (error: VerifyFailure) => !(error instanceof SettingsRefused) && error.code === 'NOT_READY' && /answered 500/.test(error.message)); + const record = w.lastRecord(); + assert.deepEqual(statuses(record.results), ['passed auth-start.e2e.ts', 'passed complete.e2e.ts', 'failed choose-organization.e2e.ts (not run)', 'failed complete-setup-mfa.e2e.ts (not run)', 'failed setup-mfa.e2e.ts (not run)']); + assert.match(record.results[4]!.error ?? '', /^an earlier group of this run failed, so this one did not run: Clerk's Platform API answered 500/); + assert.equal(w.invocations().length, 1, 'the next group would have met the same outage'); + assert.deepEqual(w.writes().filter((call) => call.startsWith('PATCH')).length, 2, 'the standard file, and the one change that failed'); + assert.ok(w.lines.some((line) => /sealed; 2 groups of 3 did not run in full/.test(line))); + }); + + it('fails a group whose settings no longer held when its specs ended, on top of what e2e reported', async () => { + const w = world(); + w.hooks.push((line) => { + if (line === `fake e2e runs ${SETUP_MFA}`) w.clerk.live()[0]!.environment['user_settings.sign_up.mfa.required'] = false; + }); + await assert.rejects(runVerb(w.deps(), RUN_ALL), (error: VerifyFailure) => error.code === 'INSTANCE_MISCONFIGURED' && error.message === `the instance no longer showed ${MFA_LABEL} when its specs ended, so what they saw is unknown`); + const record = w.lastRecord(); + assert.deepEqual(statuses(record.results).slice(3), ['passed complete-setup-mfa.e2e.ts', 'passed setup-mfa.e2e.ts', 'failed complete-setup-mfa.e2e.ts (not run)', 'failed setup-mfa.e2e.ts (not run)']); + assert.deepEqual(record.settings.map((group) => group.held), [true, true, false]); + notPublishable(w); + }); + + it('says the environment could not be read, and what answered, when that is why it cannot tell whether the settings held', async () => { + const w = world(); + w.hooks.push((line) => { + if (line === `fake e2e runs ${SETUP_MFA}`) w.frontendApi.answers = 502; + }); + const application = () => w.clerk.live()[0]!; + await assert.rejects( + runVerb(w.deps(), RUN_ALL), + (error: VerifyFailure) => error.code === 'NOT_READY' && error.message === `the instance's environment could not be read after the specs on ${MFA_LABEL} ended, so what they saw is unknown: the Frontend API of ${application().id} (${application().name}) answered 502`, + ); + const record = w.lastRecord(); + assert.deepEqual(statuses(record.results).slice(3), ['passed complete-setup-mfa.e2e.ts', 'passed setup-mfa.e2e.ts', 'failed complete-setup-mfa.e2e.ts (not run)', 'failed setup-mfa.e2e.ts (not run)']); + assert.deepEqual(record.settings.map((group) => group.held), [true, true, false]); + }); +}); + +describe('a run that finds what a daemon that failed to start left behind', () => { + const GONE = 2147483646; + const leftover = (w: ReturnType): string => join(w.workspace.agentDeviceDir, 'daemon.json'); + const plant = (w: ReturnType): void => { + mkdirSync(w.workspace.agentDeviceDir, { recursive: true }); + writeFileSync(leftover(w), JSON.stringify({ pid: GONE, version: '0.21.22' })); + writeFileSync(join(w.workspace.agentDeviceDir, 'daemon.log'), 'AGENT_DEVICE_DAEMON_PORT=50999\nDaemon error: the daemon of the warm-up gave up\n'); + }; + const driverLines = (w: ReturnType): string[] => w.lines.filter((line) => line.startsWith('driver')); + const REMOVED = `driver removed the agent-device registration of pid ${GONE}, which is not running and has no start time; a daemon failed to start, and agent-device would refuse every command until the file was gone`; + + it('removes the registration before e2e starts, so the tests run, and keeps the log of that daemon', async () => { + const w = world({ specs: { [AUTH_START]: null }, plan: { driver: { refusesALeftoverRegistration: true } } }); + plant(w); + const { record, dir } = await runVerb(w.deps(), RUN_ALL); + assert.deepEqual(statuses(record.results), ['passed auth-start.e2e.ts']); + assert.deepEqual(driverLines(w), [REMOVED]); + assert.equal(readFileSync(join(dir, 'driver', 'daemon-that-did-not-start.log'), 'utf8'), `AGENT_DEVICE_DAEMON_PORT=50999\nDaemon error: ${LEFT_OUT}\n`); + assert.match(readFileSync(join(dir, 'driver', 'summary.txt'), 'utf8'), new RegExp(`^At the start of the run daemon.json named pid ${GONE}, which was not running, and it carried no start time`)); + assert.deepEqual(record.tainted, []); + }); + + it('removes it once: when the daemon the run then starts fails too, the run fails and says what was tried', async () => { + const w = world({ plan: { driver: { refusesALeftoverRegistration: true, daemonFailsToStartIn: ['e2e'] } } }); + plant(w); + await assert.rejects(runVerb(w.deps(), RUN_ALL), (error: VerifyFailure) => { + assert.equal(error.code, 'E2E_CRASHED'); + assert.equal( + error.message, + `e2e exited 3 before writing a report for standard; agent-device's daemon did not start: it left a registration for pid 2147483645, which is not running, as the daemon before it had (pid ${GONE}), whose registration the CLI removed once in this run and does not remove a second time`, + ); + assert.match(error.fix, /e2e\.log, and driver\/summary\.txt beside it$/); + return true; + }); + assert.deepEqual(driverLines(w), [REMOVED]); + assert.deepEqual(JSON.parse(readFileSync(leftover(w), 'utf8')), { pid: 2147483645, version: '0.21.22' }, 'the second leftover is still there'); + const record = w.lastRecord(); + assert.ok(record.results.every((result) => result.status === 'failed' && result.title === 'not run')); + assert.match(record.results.at(-1)!.error!, /^e2e exited 3 before writing a report for auth_multi_factor.required_for_sign_up=true; agent-device's daemon did not start/); + }); + + it('gives the next group a clean start when the daemon fails to start during the run', async () => { + const w = world({ plan: { driver: { refusesALeftoverRegistration: true, daemonFailsToStartIn: ['e2e'] } } }); + await assert.rejects(runVerb(w.deps(), RUN_ALL), { + code: 'E2E_CRASHED', + message: "e2e exited 3 before writing a report for standard; agent-device's daemon did not start: it left a registration for pid 2147483645, which is not running", + }); + const record = w.lastRecord(); + assert.deepEqual(statuses(record.results), ['failed auth-start.e2e.ts (not run)', 'failed complete.e2e.ts (not run)', 'passed choose-organization.e2e.ts', 'passed complete-setup-mfa.e2e.ts', 'passed setup-mfa.e2e.ts']); + assert.equal(driverLines(w).length, 1); + assert.equal(readFileSync(join(w.workspace.runDir(record.run), 'driver', 'daemon-that-did-not-start.log'), 'utf8'), `AGENT_DEVICE_DAEMON_PORT=51000\nDaemon error: ${LEFT_OUT}\n`); + }); + + it('says nothing about a daemon when e2e wrote no report for another reason', async () => { + const w = world({ specs: { [AUTH_START]: null }, plan: { noReport: ['e2e'], exitCode: 1 } }); + await assert.rejects(runVerb(w.deps(), RUN_ALL), (error: VerifyFailure) => error.message === 'e2e exited 1 before writing a report for standard' && /e2e\.log$/.test(error.fix)); + assert.deepEqual(driverLines(w), []); + }); +}); + +describe('the driver logs in the evidence of a run', () => { + const SESSION = 'verify-ios-unit-0'; + const RUNNER = `sessions/${SESSION}/runner.log`; + const GATEWAY_KEY = `gw_${'k'.repeat(20)}UnitTestOnly${'z'.repeat(20)}`; + const GITHUB_TOKEN = `ghs_${'unitTestOnly'.repeat(3)}`; + const FIRST_SECRET_KEY = `sk_test_fakeSecret1${'x'.repeat(16)}`; + const typed = (seconds: string, text: string, field: string): string => ` t = ${seconds.padStart(8)}s Type '${text}' into "${field}" SecureTextField`; + const DAEMON_LINES = [ + 'Daemon error: the Backend API refused Authorization: Bearer {secretKey}', + '{"phase":"exec_command","data":{"command":"curl","args":["-H","Authorization: Bearer {platformKey}"]}}', + '{"phase":"agent","data":{"gateway":"{gatewayKey}"}}', + '{"phase":"report","data":{"github":"{githubToken}"}}', + '{"phase":"stand_in","data":{"header":"Bearer {standInToken}"}}', + '{"phase":"exec_command","command":"open","data":{"command":"xcrun","args":["simctl","launch","--terminate-running-process","UDID-1","com.clerk.E2EHost","-verifyRunId","r1","-verifySignInTicket","{ticket}"]}}', + '{"phase":"exec_command","command":"open","data":{"command":"adb","args":["-s","emulator-5554","shell","am","start","-W","--es","verifySignInTicket","{ticket}"]}}', + '{"phase":"request_failed","data":{"message":"the app answered with a session: __session={sessionToken}"}}', + '{"phase":"request_failed","data":{"message":"text entry verification failed: expected \\"{seededPassword}\\", observed \\"\\""}}', + '{"phase":"request_failed","data":{"message":"text entry verification failed: expected \\"{runPassword}\\", observed \\"\\""}}', + '{"phase":"request_failed","data":{"message":"adb -s emulator-5554 shell input text {pieces seededPassword} exited with code 1"}}', + '{"phase":"request_failed","data":{"message":"adb -s emulator-5554 shell input text {pieces runPassword} exited with code 1"}}', + ]; + const RUNNER_LINES = [ + ' t = 1.00s Find the "clerk.auth.signUp.password" SecureTextField', + typed('2.00', '{each runPassword}', 'clerk.auth.signUp.password'), + typed('3.00', '{each seededPassword}', 'clerk.auth.signIn.password'), + typed('4.00', '{pieces runPassword}', 'clerk.auth.signUp.password'), + typed('5.00', '{pieces seededPassword}', 'clerk.auth.signIn.password'), + ' t = 6.00s Find the "{sessionToken}" StaticText', + 'TextField, 0x1, {{0.0, 0.0}, {100.0, 44.0}}, identifier: \'ticket\', value: {ticket}', + ' t = 7.00s Synthesize event', + ]; + const KINDS = ['secretKey', 'platformKey', 'gatewayKey', 'githubToken', 'standInToken', 'ticket', 'sessionToken', 'seededPassword', 'runPassword'] as const; + const TYPED = ['seededPassword', 'runPassword'] as const; + type Kind = (typeof KINDS)[number]; + + const piecesOf = (password: string): string[] => password.match(/.{1,16}/g)!.filter((piece) => piece.length >= 8); + const quotedAlone = (text: string): string => [...text.matchAll(/(['"])(.)\1/g)].map((match) => match[2]).join(''); + const runWith = async (writes: Readonly>) => { + const w = world({ specs: { [AUTH_START]: null }, plan: { driver: { known: { secretKey: FIRST_SECRET_KEY, platformKey: PLATFORM_KEY, gatewayKey: GATEWAY_KEY, githubToken: GITHUB_TOKEN }, writes } } }); + const { record, dir } = await runVerb({ ...w.deps(), env: { GITHUB_TOKEN }, agent: () => readAgent({ AI_GATEWAY_API_KEY: GATEWAY_KEY }) }, RUN_ALL); + const secrets = JSON.parse(readFileSync(join(w.dir, 'secrets-the-driver-saw.json'), 'utf8')) as Record; + assert.equal(secrets.secretKey, w.clerk.live()[0]!.sk, 'the key planted is the key of the application the run used'); + assert.equal(secrets.runPassword, `Verify-${record.run}-Pw1!`); + return { w, record, dir, secrets }; + }; + + it('holds no secret of any kind, whole, in the pieces a test types it in, or spelled a character at a time', async () => { + const { record, dir, secrets } = await runWith({ 'daemon.log': DAEMON_LINES, [RUNNER]: RUNNER_LINES }); + const copies = { daemon: readFileSync(join(dir, 'driver', 'daemon.log'), 'utf8'), runner: readFileSync(join(dir, 'driver', `runner-${SESSION}.log`), 'utf8'), summary: readFileSync(join(dir, 'driver', 'summary.txt'), 'utf8') }; + for (const [name, copy] of Object.entries(copies)) { + for (const kind of KINDS) assert.equal(copy.includes(secrets[kind]), false, `the ${name} holds the ${kind}`); + for (const kind of TYPED) { + for (const piece of piecesOf(secrets[kind])) assert.equal(copy.includes(piece), false, `the ${name} holds a typed piece of the ${kind}`); + assert.equal(quotedAlone(copy).includes(secrets[kind].slice(0, 8)), false, `the ${name} spells the ${kind}`); + } + } + const typedLines = [secrets.runPassword.length, secrets.seededPassword.length, 3, 2]; + assert.deepEqual(copies.daemon.trim().split('\n'), [`Daemon error: ${LEFT_OUT}`, linesLeftOut(9 + 3 + 2)]); + assert.deepEqual(copies.runner.trimEnd().split('\n'), [ + ` t = 1.00s Find the "${LEFT_OUT}" SecureTextField`, + ...['2.00', '3.00', '4.00', '5.00'].flatMap((seconds, at) => Array.from({ length: typedLines[at]! }, () => ` t = ${seconds.padStart(8)}s Type ${LEFT_OUT}`)), + ` t = 6.00s Find the "${LEFT_OUT}" StaticText`, + linesLeftOut(1), + ' t = 7.00s Synthesize event', + ]); + assert.deepEqual(record.tainted, [], 'nothing the run kept holds a secret, so the workflow uploads it'); + assert.doesNotThrow(() => assertPublishable(record, [])); + }); + + it('is tainted by a log that reached the run directory with a secret of any kind in it, so nothing of the run is uploaded or attached', async () => { + const unredacted: Readonly> = { + 'whole-secretKey': ['Bearer {secretKey}'], + 'whole-platformKey': ['Bearer {platformKey}'], + 'whole-gatewayKey': ['{gatewayKey}'], + 'whole-githubToken': ['{githubToken}'], + 'whole-standInToken': ['Bearer {standInToken}'], + 'whole-ticket': ['"-verifySignInTicket","{ticket}"'], + 'whole-sessionToken': ['__session={sessionToken}'], + 'whole-seededPassword': ['expected "{seededPassword}"'], + 'whole-runPassword': ['expected "{runPassword}"'], + 'pieces-seededPassword': ['input text {pieces seededPassword}'], + 'pieces-runPassword': ['input text {pieces runPassword}'], + 'spelled-seededPassword': [typed('1.00', '{each seededPassword}', 'clerk.auth.signIn.password')], + 'spelled-runPassword': [typed('1.00', '{each runPassword}', 'clerk.auth.signUp.password')], + }; + const { record, dir } = await runWith({ + ...Object.fromEntries(Object.entries(unredacted).map(([name, lines]) => [`RUN/driver/${name}.log`, lines])), + 'RUN/driver/nothing-secret.log': [' t = 1.00s Find the "clerk.auth.signUp.password" SecureTextField', "Type 'a' into a field that is not a secret"], + }); + assert.deepEqual(record.tainted, Object.keys(unredacted).map((name) => join(dir, 'driver', `${name}.log`)).sort()); + assert.throws(() => assertPublishable(record, []), (error: VerifyFailure) => error.code === 'EVIDENCE_UNSAFE' && /has secret values in/.test(error.message)); + }); + + it('is sealed after the driver logs are copied, so a copy that held a secret would taint the run', async () => { + const w = world({ specs: { [AUTH_START]: null } }); + const copiesASecret: Deps['watchDriver'] = (stateDir, runDir, ...rest) => { + const watch = watchDriver(stateDir, runDir, ...rest); + return { + ...watch, + collect: () => { + watch.collect(); + writeFileSync(join(runDir, 'driver', 'runner-that-kept-a-secret.log'), `Verify-${runDir.split('/').at(-1)}-Pw1!\n`); + }, + }; + }; + const { record, dir } = await runVerb({ ...w.deps(), watchDriver: copiesASecret }, RUN_ALL); + assert.deepEqual(record.tainted, [join(dir, 'driver', 'runner-that-kept-a-secret.log')]); + assert.throws(() => assertPublishable(record, []), (error: VerifyFailure) => error.code === 'EVIDENCE_UNSAFE'); + }); + + it('keeps only what the logs gained during the run, and the whole of a log a new daemon rewrote', async () => { + const before = (w: ReturnType): void => { + mkdirSync(join(w.workspace.agentDeviceDir, 'sessions', SESSION), { recursive: true }); + writeFileSync(join(w.workspace.agentDeviceDir, 'daemon.log'), 'AGENT_DEVICE_DAEMON_PORT=50999\n'); + writeFileSync(join(w.workspace.agentDeviceDir, RUNNER), ' t = 1.00s Set Up\n'); + }; + const w = world({ specs: { [AUTH_START]: null }, plan: { driver: { rewrites: ['daemon.log'], writes: { 'daemon.log': ['AGENT_DEVICE_DAEMON_PORT=51000'], [RUNNER]: [' t = 2.00s Tear Down'] } } } }); + before(w); + const { dir } = await runVerb(w.deps(), RUN_ALL); + assert.equal(readFileSync(join(dir, 'driver', 'daemon.log'), 'utf8'), 'AGENT_DEVICE_DAEMON_PORT=51000\n'); + assert.equal(readFileSync(join(dir, 'driver', `runner-${SESSION}.log`), 'utf8'), ' t = 2.00s Tear Down\n'); + }); + + it('says in the evidence that no daemon wrote a log, when none did', async () => { + const w = world({ specs: { [AUTH_START]: null } }); + const { record, dir } = await runVerb(w.deps(), RUN_ALL); + assert.deepEqual(readdirSync(join(dir, 'driver')), ['summary.txt']); + assert.match(readFileSync(join(dir, 'driver', 'summary.txt'), 'utf8'), /\nThere is no daemon\.log, so no agent-device daemon on this machine wrote one\.\n/); + assert.deepEqual(record.tainted, []); + }); +}); diff --git a/integration/expo-native/test/seam.test.ts b/integration/expo-native/test/seam.test.ts new file mode 100644 index 00000000000..735368e0cc7 --- /dev/null +++ b/integration/expo-native/test/seam.test.ts @@ -0,0 +1,101 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { mkdirSync, mkdtempSync, readFileSync, readdirSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join, relative, resolve, sep } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { describe, it } from 'node:test'; + +const PACKAGE_DIR = join(dirname(fileURLToPath(import.meta.url)), '..'); + +const QUOTED_PATH = /(['"`])(\.{1,2}\/[^'"`\n]*)\1/g; + +function filesUnder(dir: string, skips: (name: string) => boolean = () => false): string[] { + return readdirSync(dir, { withFileTypes: true }).flatMap((entry) => { + const path = join(dir, entry.name); + if (entry.isDirectory()) return skips(entry.name) ? [] : filesUnder(path, skips); + return entry.isFile() ? [path] : []; + }); +} + +const posix = (path: string): string => path.split(sep).join('/'); + +export function pathsThatLeaveTheTests(packageDir: string): readonly string[] { + const specs = join(packageDir, 'specs'); + const config = join(packageDir, 'e2e.config.ts'); + const inside = (file: string): boolean => file === config || file === specs || file.startsWith(`${specs}${sep}`); + return [config, ...filesUnder(specs)].flatMap((file) => + [...readFileSync(file, 'utf8').matchAll(QUOTED_PATH)] + .map((match) => resolve(dirname(file), match[2]!)) + .filter((target) => !inside(target)) + .map((target) => `${posix(relative(packageDir, file))} names ${posix(relative(packageDir, target))}`), + ); +} + +describe('the tests are an e2e project of their own', () => { + it('names no path outside specs/ in e2e.config.ts or in any file under specs/, so the CLI can be deleted and `npx e2e run` still loads (not seen: a symbolic link, a "#" import alias, and a path that is computed)', () => { + assert.deepEqual(pathsThatLeaveTheTests(PACKAGE_DIR), []); + }); + + it('reads e2e.config.ts and every file under specs/, whatever its extension and however deep', () => { + const packageDir = mkdtempSync(join(tmpdir(), 'verify-seam-')); + const files: Readonly> = { + 'e2e.config.ts': "import { app } from './specs/app.ts';\nimport './src/core/keys.ts';\n", + 'specs/app.ts': "export { app } from './support/inputs.ts';\n", + 'specs/support/inputs.cts': "import keys = require('../../src/core/keys.ts');\n", + 'specs/golden/sign-in/complete.e2e.tsx': "import '../../fixtures.ts';\nexport const load = (name: string) => import(`../../../testing/${name}.ts`);\n", + 'specs/golden/sign-in/complete.settings.json': '{ "extends": "../../../src/core/instances/base.json" }\n', + 'src/core/keys.ts': "import '../../specs/support/inputs.cts';\n", + }; + for (const [file, text] of Object.entries(files)) { + mkdirSync(dirname(join(packageDir, file)), { recursive: true }); + writeFileSync(join(packageDir, file), text); + } + assert.deepEqual([...pathsThatLeaveTheTests(packageDir)].sort(), [ + 'e2e.config.ts names src/core/keys.ts', + 'specs/golden/sign-in/complete.e2e.tsx names testing/${name}.ts', + 'specs/golden/sign-in/complete.settings.json names src/core/instances/base.json', + 'specs/support/inputs.cts names src/core/keys.ts', + ]); + }); + + it('sees a relative path however a file names it: in an import, a re-export, a dynamic or commented import, a template, require under any name, import.meta.resolve, and a URL', () => { + const lines = [ + "import { x } from '../src/core/x.ts';", + "export type { X } from '../../src/core/types.ts';", + 'export * from "../src/core/types.ts";', + "await import('../src/host.ts');", + "await import(/* webpackIgnore: true */ '../src/host.ts');", + 'await import(`../src/core/${name}.ts`);', + "import '../testing/git-env.ts';", + "require('../src/core/launch.mjs');", + "requireOnlyForAnAgent('../src/core/agent.ts');", + "createRequire(import.meta.url)('../src/core/agent.ts');", + "import.meta.resolve('../src/core/keys.ts');", + "new URL('../bin/control', import.meta.url);", + '{ "extends": "../src/core/instances/base.json" }', + ]; + for (const line of lines) assert.equal([...line.matchAll(QUOTED_PATH)].length, 1, line); + assert.equal([..."import { test } from '@e2e-dev/mobile'; import { join } from 'node:path'; join(here, '..', '..');".matchAll(QUOTED_PATH)].length, 0, 'a package and a path segment are not relative paths'); + }); +}); + +const KEY_PREFIXES = ['sk_test_', 'sk_live_', 'rk_test_', 'rk_live_']; +const SCANNED_KEY = new RegExp(`(? { + it('never have a length that a secret scanner takes for a real key: exactly 24 letters and digits after the prefix, which GitHub refuses to take in a push, or 40 and more', () => { + const held = filesUnder(PACKAGE_DIR, (name) => name === 'node_modules' || name.startsWith('.')) + .filter((file) => SCANNED_KEY.test(readFileSync(file, 'utf8'))) + .map((file) => posix(relative(PACKAGE_DIR, file))); + assert.deepEqual(held, [], 'give the made-up key in each of these files another length'); + }); + + it('are told from scanned ones by their length alone', () => { + for (const prefix of KEY_PREFIXES) { + for (const length of [24, 40, 42, 99]) assert.equal(SCANNED_KEY.test(`const key = '${prefix}${'a1B2'.repeat(25).slice(0, length)}';`), true, `${prefix} and ${length}`); + for (const length of [12, 23, 25, 32, 39]) assert.equal(SCANNED_KEY.test(`const key = '${prefix}${'a1B2'.repeat(25).slice(0, length)}';`), false, `${prefix} and ${length}`); + } + assert.equal(SCANNED_KEY.test(`pk_test_${'a'.repeat(24)}`), false, 'a publishable key is public'); + }); +}); diff --git a/integration/expo-native/test/secrets.test.ts b/integration/expo-native/test/secrets.test.ts new file mode 100644 index 00000000000..9f4814cb156 --- /dev/null +++ b/integration/expo-native/test/secrets.test.ts @@ -0,0 +1,104 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { randomBytes } from 'node:crypto'; +import { existsSync, mkdtempSync, rmSync, truncateSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { sealEvidence } from '../src/core/evidence.ts'; +import type { EvidencePath } from '../src/core/types.ts'; +import { holdsJwt, protect, redact } from '../specs/support/secret.ts'; + +const part = (claims: object): string => Buffer.from(JSON.stringify(claims)).toString('base64url'); +const newJwt = (): string => [part({ alg: 'RS256', typ: 'JWT' }), part({ sid: `sess_${randomBytes(4).toString('hex')}` }), randomBytes(32).toString('base64url')].join('.'); +const NO_RECORD = {} as Parameters[1]; + +function sealed(files: Readonly>, secrets: readonly string[]): readonly string[] { + const dir = mkdtempSync(join(tmpdir(), 'verify-secrets-')) as EvidencePath; + for (const [name, content] of Object.entries(files)) writeFileSync(join(dir, name), content); + return sealEvidence(dir, NO_RECORD, secrets).tainted.map((file) => file.slice(dir.length + 1)); +} + +describe('redact', () => { + it('takes out a token shaped like a JWT though no one told it the value', () => { + const jwt = newJwt(); + assert.equal(redact(`Cookie: __session=${jwt}; __client=${newJwt()}`), 'Cookie: __session=; __client='); + assert.equal(redact(`"Authorization":"Bearer ${jwt}"`), '"Authorization":"Bearer "'); + }); + + it('takes out a JWT that has other text stuck to its front, and only the JWT', () => { + const jwt = newJwt(); + assert.equal(redact(`?ticket%3D${jwt}&next=home`), '?ticket%3D&next=home'); + assert.equal(redact(`${jwt}.${'x'.repeat(20)} ${jwt}`), `.${'x'.repeat(20)} `); + }); + + it('reads a megabyte of text that could be the start of a JWT at every place in less than a second', { timeout: 5_000 }, () => { + const started = Date.now(); + const almost = 'eyJ'.repeat(350_000); + assert.equal(redact(almost), almost); + assert.equal(holdsJwt(Buffer.from(`${almost}.eyJ`)), false); + assert.ok(Date.now() - started < 1_000, `${Date.now() - started} ms`); + }); + + it('takes out each JWT of one unbroken run of characters that holds two hundred thousand of them', { timeout: 30_000 }, () => { + const jwt = newJwt(); + const many = Array.from({ length: 200_000 }, () => jwt); + assert.equal(holdsJwt(Buffer.from(many.join('.'))), true); + assert.equal(redact(many.join('.')), many.map(() => '').join('.')); + }); + + it('leaves text alone that only looks a little like a JWT', () => { + for (const text of ['eyJhbGciOiJSUzI1NiJ9', 'eyJhbGciOiJSUzI1NiJ9.notapayloadatall.signaturesignature', 'a.b.c', 'pk_test_ZmFrZS0xLmNsZXJrLmFjY291bnRzLmRldiQ', 'specs/golden/sign-up/complete.e2e.ts']) { + assert.equal(redact(text), text); + } + }); + + it('takes out the longer of two values first when one holds the other', () => { + const inner = `inner-${randomBytes(8).toString('hex')}`; + const outer = `outer-${inner}-end`; + protect(inner); + protect(outer); + assert.equal(redact(`saw ${outer} and ${inner}`), 'saw and '); + }); +}); + +describe('the seal check', () => { + it('taints a file that holds a JWT, a value the CLI never held', () => { + assert.deepEqual(sealed({ 'driver.log': `__session=${newJwt()}`, 'clean.log': 'eyJhbGciOiJSUzI1NiJ9 is half a header' }, []), ['driver.log']); + }); + + it('finds a JWT that starts near the end of one stretch of a large file and ends in the next', () => { + const jwt = newJwt(); + const stretch = 8 * 1024 * 1024; + for (const before of [stretch - 20, stretch - 64 * 1024 + 5, 2 * stretch - 64 * 1024 - 30]) { + assert.equal(holdsJwt(Buffer.concat([Buffer.alloc(before, 0x20), Buffer.from(jwt)])), true, `a JWT ${before} bytes in`); + } + assert.equal(holdsJwt(Buffer.alloc(2 * stretch, 0x65)), false); + }); + + it('taints a file that spells a secret one quoted character at a time, as XCTest shows typed text', () => { + const password = `${randomBytes(12).toString('hex')}Aa1!`; + const typedLines = (quote: string): string => [...password].map((character, at) => ` t = ${at}.00s Type ${quote}${character}${quote} into "password" SecureTextField`).join('\n'); + assert.deepEqual( + sealed({ 'single.log': typedLines("'"), 'double.log': typedLines('"'), 'escaped.log': typedLines('\\"'), 'other.log': typedLines("'").replaceAll(password[3]!, password[3] === 'z' ? 'y' : 'z'), 'none.log': "Type 'a' into a field" }, [password]), + ['double.log', 'escaped.log', 'single.log'], + ); + }); + + it('does not seal a run when a file of it cannot be scanned, so the run has no run.json and nothing of it is uploaded', () => { + const dir = mkdtempSync(join(tmpdir(), 'verify-secrets-')) as EvidencePath; + try { + writeFileSync(join(dir, 'app.log'), 'nothing secret'); + writeFileSync(join(dir, 'too-large-to-read.log'), ''); + truncateSync(join(dir, 'too-large-to-read.log'), 2 ** 31); + assert.throws(() => sealEvidence(dir, NO_RECORD, []), { code: 'ERR_FS_FILE_TOO_LARGE' }); + assert.equal(existsSync(join(dir, 'run.json')), false); + } finally { + rmSync(dir, { recursive: true }); + } + }); + + it('does not take a secret shorter than eight characters for a needle, whole or spelled', () => { + assert.deepEqual(sealed({ 'short.log': "the code 424242 and 'A''a''1''!'" }, ['424242', 'Aa1!']), []); + }); +}); diff --git a/integration/expo-native/test/settings.test.ts b/integration/expo-native/test/settings.test.ts new file mode 100644 index 00000000000..cd2740df80a --- /dev/null +++ b/integration/expo-native/test/settings.test.ts @@ -0,0 +1,249 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { mkdirSync, mkdtempSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { STANDARD, STANDARD_FILE, configFor, configLeaves, declaredIn, expectedEnvironment, planGroups, readSpecText, sameLeaf, settingsFileOf, settingsOf, sourceHash, standardFile, straySettingsFile } from '../src/core/instances/settings.ts'; +import { VerifyFailure, type InstanceSettings, type SpecRef } from '../src/core/types.ts'; + +const MFA: InstanceSettings = { config: { auth_multi_factor: { required_for_sign_up: true } }, environment: { 'user_settings.sign_up.mfa.required': true } }; +const FORCED_ORG: InstanceSettings = { config: { organization_settings: { force_organization_selection: true } }, environment: { 'organization_settings.force_organization_selection': true } }; + +const SPEC_SOURCE = "import { test } from '../../fixtures.ts';\n\ntest('x', async ({ host }) => { await host.launch({}); });\n"; +const spec = (path: string): SpecRef => ({ kind: 'golden', path, feature: null }); +const text = (declaration: InstanceSettings | string | null, source = SPEC_SOURCE) => ({ source, declaration: declaration === null || typeof declaration === 'string' ? declaration : JSON.stringify(declaration) }); +const file = (path: string, declared: InstanceSettings | string | null = null) => ({ spec: spec(path), ...text(declared) }); +const usage = (pattern: RegExp) => (error: VerifyFailure) => error instanceof VerifyFailure && error.code === 'USAGE' && pattern.test(error.message); +const ODD = 'specs/explored/odd.e2e.ts'; +const ODD_FILE = 'specs/explored/odd.settings.json'; +const refuses = (declarations: Readonly>, why: RegExp = /./): void => { + for (const [what, declaration] of Object.entries(declarations)) { + assert.throws( + () => declaredIn(text(declaration), ODD), + (error: VerifyFailure) => error instanceof VerifyFailure && error.code === 'USAGE' && error.message.startsWith(`${ODD_FILE}: `) && why.test(error.message) && error.fix.startsWith(`write the settings of ${ODD} as one JSON object, as in \`{ "config": `), + what, + ); + } +}; + +describe('the standard file', () => { + it('pins a standard value for every setting the two golden declarations change', () => { + const pinned = configLeaves(standardFile().config); + assert.equal(pinned['auth_multi_factor.required_for_sign_up'], false); + assert.equal(pinned['organization_settings.force_organization_selection'], false); + assert.deepEqual(pinned['auth_email.sign_in_strategies'], ['email_code', 'email_link'], 'a list is one leaf'); + }); +}); + +describe('reading a declaration from the settings file beside a spec', () => { + it('names the settings file after the spec file', () => { + assert.equal(settingsFileOf('specs/golden/session-tasks/setup-mfa.e2e.ts'), 'specs/golden/session-tasks/setup-mfa.settings.json'); + assert.equal(settingsFileOf(ODD), ODD_FILE); + }); + + it('reads the spec and the file beside it, and nothing when the spec has no file', () => { + const dir = mkdtempSync(join(tmpdir(), 'verify-settings-')); + mkdirSync(join(dir, 'specs/explored'), { recursive: true }); + writeFileSync(join(dir, ODD), SPEC_SOURCE); + assert.deepEqual(readSpecText(dir, ODD), { source: SPEC_SOURCE, declaration: null }); + assert.equal(declaredIn(readSpecText(dir, ODD), ODD), null); + writeFileSync(join(dir, ODD_FILE), `${JSON.stringify(MFA, null, 2)}\n`); + assert.deepEqual(declaredIn(readSpecText(dir, ODD), ODD), MFA); + }); + + it('reads config and environment, with nesting, lists, numbers, and null', () => { + const declared = { config: { auth_password: { min_length: 12, max_length: -1, enabled: false, note: null }, auth_email: { sign_in_strategies: ['email_code', 'email_link'] } }, environment: { 'user_settings.password_settings.min_length': 12, 'auth_config.second_factors': ['totp'] } }; + assert.deepEqual(declaredIn(text(declared), ODD), declared); + }); + + it('fails closed, with the file and the form to write, on anything that is not one JSON object with a config and an environment', () => { + refuses( + { + 'an empty file': '', + 'a JavaScript literal': "{ config: { auth_multi_factor: { required_for_sign_up: true } }, environment: { 'user_settings.sign_up.mfa.required': true } }", + 'a trailing comma': '{ "config": { "a": { "b": true } }, "environment": { "a.b": true }, }', + 'a comment': `// why\n${JSON.stringify(MFA)}`, + 'an export': `export default ${JSON.stringify(MFA)}`, + }, + /it is not JSON \(/, + ); + refuses({ 'a list': '[1, 2]', 'a string': '"mfa"', null: 'null' }, /it is not a JSON object/); + refuses({ 'a key other than config and environment': '{ "config": { "a": { "b": true } }, "environment": { "a.b": true }, "extra": 1 }' }, /it has extra, and only config and environment are read/); + refuses({ 'no config': '{ "environment": { "a.b": true } }', 'an empty config': '{ "config": {}, "environment": { "a.b": true } }', 'a config that is a list': '{ "config": [], "environment": { "a.b": true } }' }, /it needs a config object/); + refuses({ 'no environment': '{ "config": { "a": { "b": true } } }', 'an empty environment': '{ "config": { "a": { "b": true } }, "environment": {} }' }, /it needs an environment object/); + refuses( + { 'an environment value that is an object': '{ "config": { "a": { "b": true } }, "environment": { "a": { "b": true } } }', 'an environment list of objects': '{ "config": { "a": { "b": true } }, "environment": { "a.b": [{ "c": 1 }] } }' }, + /environment leaf a(\.b)? is an object; write each leaf by its full dotted path/, + ); + }); + + it('gives a spec a new hash when its settings file changes, appears, or goes, so a run can tell that its plan is stale', () => { + const hashes = [text(null), text(MFA), text(FORCED_ORG), text(''), text(MFA, `${SPEC_SOURCE}\n`)].map(sourceHash); + assert.equal(new Set(hashes).size, hashes.length); + assert.equal(sourceHash(text(MFA)), sourceHash(text(MFA))); + assert.match(hashes[0]!, /^[0-9a-f]{64}$/); + }); + + it('finds a JSON file under specs that is not the settings file of a spec beside it, and none when each has its spec', () => { + const dir = mkdtempSync(join(tmpdir(), 'verify-settings-')); + assert.equal(straySettingsFile(dir), null, 'a package with no specs directory has none'); + mkdirSync(join(dir, 'specs/golden/session-tasks'), { recursive: true }); + for (const name of ['setup-mfa.e2e.ts', 'setup-mfa.settings.json']) writeFileSync(join(dir, 'specs/golden/session-tasks', name), ''); + assert.equal(straySettingsFile(dir), null); + writeFileSync(join(dir, 'specs/golden/session-tasks/setup-mfa.e2e.settings.json'), ''); + const stray = straySettingsFile(dir); + assert.equal(stray?.code, 'USAGE'); + assert.equal(stray?.message, 'specs/golden/session-tasks/setup-mfa.e2e.settings.json is not the settings file of a spec, so no run reads it'); + assert.equal(stray?.fix, 'name a settings file after its spec file, with .settings.json in place of .e2e.ts, or delete it'); + for (const misnamed of ['setup-mfa.setting.json', 'setup-mfa-settings.json', 'settings.json', 'setup-mfa.settings.jsonc']) { + const other = mkdtempSync(join(tmpdir(), 'verify-settings-')); + mkdirSync(join(other, 'specs/explored'), { recursive: true }); + for (const name of ['setup-mfa.e2e.ts', misnamed]) writeFileSync(join(other, 'specs/explored', name), ''); + assert.equal(straySettingsFile(other)?.message, `specs/explored/${misnamed} is not the settings file of a spec, so no run reads it`); + } + }); +}); + +describe('checking a declaration against the standard file', () => { + it('names the settings by the config leaves they change', () => { + const settings = settingsOf(MFA, 'specs/a.e2e.ts'); + assert.equal(settings.label, 'auth_multi_factor.required_for_sign_up=true'); + assert.equal(settings.askedBy, 'specs/a.e2e.ts'); + assert.notEqual(settings.key, STANDARD.key); + assert.equal(settingsOf({ config: { auth_password: { min_length: 12 }, auth_email: { sign_in_strategies: ['email_code'] } }, environment: { 'user_settings.password_settings.min_length': 12 } }, 'a').label, 'auth_password.min_length=12, auth_email.sign_in_strategies=["email_code"]'); + assert.deepEqual(STANDARD, { key: STANDARD.key, label: 'standard', declared: null, askedBy: null }); + assert.equal(settingsOf(null, 'specs/a.e2e.ts'), STANDARD); + }); + + it('sends the whole standard config with the declaration laid over it', () => { + const body = configFor(settingsOf(MFA, 'a')); + const standard = standardFile().config; + assert.deepEqual(Object.keys(body), Object.keys(standard)); + assert.deepEqual(body.auth_multi_factor, { ...(standard.auth_multi_factor as object), required_for_sign_up: true }); + assert.deepEqual(body.organization_settings, standard.organization_settings, 'so a setting an earlier group changed goes back to standard'); + assert.equal(configFor(STANDARD), standard); + }); + + it('gives two declarations with one effect one key, and different effects different keys', () => { + const alsoStandardLeaf: InstanceSettings = { config: { auth_multi_factor: { required_for_sign_up: true, required_for_sign_in: false } }, environment: MFA.environment }; + assert.equal(settingsOf(alsoStandardLeaf, 'b').key, settingsOf(MFA, 'a').key); + assert.notEqual(settingsOf(FORCED_ORG, 'c').key, settingsOf(MFA, 'a').key); + }); + + it('refuses a config leaf the standard file does not pin, and says what to add', () => { + const unpinned: InstanceSettings = { config: { auth_email: { brand_new_toggle: true } }, environment: MFA.environment }; + assert.throws(() => settingsOf(unpinned, 'specs/a.e2e.ts'), (error: VerifyFailure) => error.code === 'INSTANCE_MISCONFIGURED' && error.message.includes('specs/a.e2e.ts') && error.message.includes('auth_email.brand_new_toggle') && error.fix.startsWith("check the spelling against Clerk's Platform API config; ") && error.fix.includes(`; add the standard value of \`auth_email.brand_new_toggle\` to \`config\` in ${STANDARD_FILE}`)); + }); + + it('lists the keys the standard file has beside a config key it does not, before it says to change the file', () => { + const fixOf = (config: InstanceSettings['config']): string => { + try { + settingsOf({ config, environment: MFA.environment }, 'specs/a.e2e.ts'); + } catch (error) { + return (error as VerifyFailure).fix; + } + return assert.fail('the declaration was accepted'); + }; + const add = (leaf: string) => `add the standard value of \`${leaf}\` to \`config\` in ${STANDARD_FILE}`; + assert.ok(fixOf({ auth_multi_factor: { requird_for_sign_up: true } }).startsWith(`check the spelling against Clerk's Platform API config; the standard file has authenticator_app, backup_code, required_for_sign_in, required_for_sign_up under \`auth_multi_factor\`; ${add('auth_multi_factor.requird_for_sign_up')}`)); + assert.ok(fixOf({ auth_multi_factor: { backup_code: { enabld: false } } }).startsWith(`check the spelling against Clerk's Platform API config; the standard file has enabled under \`auth_multi_factor.backup_code\`; ${add('auth_multi_factor.backup_code.enabld')}`)); + const top = Object.keys(standardFile().config); + assert.ok(top.length > 12); + assert.ok(fixOf({ auth_multi_factr: { required_for_sign_up: true } }).startsWith(`check the spelling against Clerk's Platform API config; the standard file has ${top.slice(0, 12).join(', ')}, and ${top.length - 12} more at the top of \`config\`; ${add('auth_multi_factr.required_for_sign_up')}`), 'a misspelt parent lists what stands beside it, and never more than 12'); + assert.ok(fixOf({ auth_multi_factor: { required_for_sign_up: { always: true } } }).startsWith(`check the spelling against Clerk's Platform API config; ${add('auth_multi_factor.required_for_sign_up.always')}`), 'nothing to list under a key that holds a plain value'); + }); + + it('says the standard file has an object where a declaration gives one plain value, and names its keys', () => { + assert.throws( + () => settingsOf({ config: { auth_multi_factor: true }, environment: MFA.environment }, 'specs/a.e2e.ts'), + (error: VerifyFailure) => + error.code === 'INSTANCE_MISCONFIGURED' && + error.message === 'specs/a.e2e.ts declares auth_multi_factor as true, and the standard file has an object there' && + error.fix === 'declare the keys of `auth_multi_factor` to change, each by its own name; the standard file has authenticator_app, backup_code, required_for_sign_in, required_for_sign_up there', + ); + }); + + it('compares the members of a list by what they hold, in any order', () => { + assert.equal(sameLeaf([{ a: 1 }], [{ a: 2 }]), false, 'two lists of objects of one length are not the same list'); + assert.equal(sameLeaf([{ a: 1, b: [2, 3] }, { c: null }], [{ c: null }, { b: [2, 3], a: 1 }]), true); + assert.equal(sameLeaf(['b', 'a'], ['a', 'b']), true); + assert.equal(sameLeaf(['a'], ['a', 'a']), false); + const declared = settingsOf({ config: MFA.config, environment: { 'auth_config.second_factors': ['totp', 'phone_code'] } }, 'a'); + assert.deepEqual(expectedEnvironment(declared)['auth_config.second_factors'], ['phone_code', 'totp'], 'a declared list is expected in the order a live one is read in'); + }); + + it('refuses a declaration that only restates standard values', () => { + assert.throws(() => settingsOf({ config: { auth_multi_factor: { required_for_sign_up: false } }, environment: MFA.environment }, 'specs/a.e2e.ts'), (error: VerifyFailure) => usage(/^specs\/a\.e2e\.ts declares only standard values/)(error) && error.fix.startsWith('delete specs/a.settings.json; ')); + }); + + it('refuses an environment leaf at its standard value, and takes one the standard file does not list', () => { + assert.throws(() => settingsOf({ config: MFA.config, environment: { 'user_settings.sign_up.mfa.required': false } }, 'specs/a.e2e.ts'), usage(/user_settings\.sign_up\.mfa\.required to be false, which is its standard value/)); + const listed = standardFile().environment['auth_config.first_factors'] as readonly string[]; + assert.throws(() => settingsOf({ config: MFA.config, environment: { 'auth_config.first_factors': [...listed].reverse() } }, 'a'), usage(/standard value/), 'a list in another order is the same value'); + assert.ok(settingsOf({ config: MFA.config, environment: { 'auth_config.reverification': false } }, 'a'), 'the instance decides whether a leaf the file does not list shows the change'); + }); + + it('does not take a name every object inherits for a pinned config leaf', () => { + assert.throws(() => settingsOf({ config: { constructor: true }, environment: MFA.environment }, 'specs/a.e2e.ts'), (error: VerifyFailure) => error.code === 'INSTANCE_MISCONFIGURED' && /^specs\/a\.e2e\.ts declares constructor, and the standard file has no value/.test(error.message)); + }); + + it('requires that a user may delete their own account', () => { + assert.equal(standardFile().environment['user_settings.actions.delete_self'], true); + }); + + it('expects the required leaves of the standard file, with the declared leaves laid over them and added to them', () => { + const required = Object.keys(standardFile().environment); + const expected = expectedEnvironment(settingsOf(MFA, 'a')); + assert.deepEqual(Object.keys(expected), required); + assert.equal(expected['user_settings.sign_up.mfa.required'], true); + assert.equal(expectedEnvironment(STANDARD)['user_settings.sign_up.mfa.required'], false); + const unlisted = expectedEnvironment(settingsOf({ config: MFA.config, environment: { 'auth_config.reverification': false } }, 'a')); + assert.deepEqual([Object.keys(unlisted).length, unlisted['auth_config.reverification']], [required.length + 1, false]); + }); +}); + +describe('planning a run in groups', () => { + const golden = [file('specs/golden/auth-start/auth-start.e2e.ts'), file('specs/golden/session-tasks/choose-organization.e2e.ts', FORCED_ORG), file('specs/golden/session-tasks/complete-setup-mfa.e2e.ts', MFA), file('specs/golden/session-tasks/setup-mfa.e2e.ts', MFA), file('specs/golden/sign-up/complete.e2e.ts')]; + const mfaKey = settingsOf(MFA, 'x').key; + const orgKey = settingsOf(FORCED_ORG, 'x').key; + const order = (applied: string | null, specs = golden) => planGroups(specs, applied).map((group) => group.settings.label); + + it('puts the group already applied first, so a run changes the instance once less than it has groups', () => { + assert.deepEqual(order(STANDARD.key), ['standard', 'organization_settings.force_organization_selection=true', 'auth_multi_factor.required_for_sign_up=true']); + assert.deepEqual(order(mfaKey), ['auth_multi_factor.required_for_sign_up=true', 'organization_settings.force_organization_selection=true', 'standard']); + assert.deepEqual(order(orgKey), ['organization_settings.force_organization_selection=true', 'auth_multi_factor.required_for_sign_up=true', 'standard']); + }); + + it('with nothing applied, or settings no group has, keeps the order the specs came in and leaves standard for last', () => { + for (const applied of [null, 'ffffffffffff']) assert.deepEqual(order(applied), ['organization_settings.force_organization_selection=true', 'auth_multi_factor.required_for_sign_up=true', 'standard']); + }); + + it('is stable: the same files in the same order plan the same groups, each with its files in order', () => { + const plan = planGroups(golden, STANDARD.key); + assert.deepEqual(plan, planGroups(golden, STANDARD.key)); + assert.deepEqual(plan.map((group) => group.specs.map((s) => s.path.split('/').at(-1))), [['auth-start.e2e.ts', 'complete.e2e.ts'], ['choose-organization.e2e.ts'], ['complete-setup-mfa.e2e.ts', 'setup-mfa.e2e.ts']]); + assert.equal(plan[2]!.settings.askedBy, 'specs/golden/session-tasks/complete-setup-mfa.e2e.ts', 'the first spec of a group is the one that asked'); + assert.equal(plan[0]!.settings, STANDARD); + assert.match(plan[0]!.specs[0]!.sourceHash, /^[0-9a-f]{64}$/); + assert.notEqual(plan[0]!.specs[0]!.sourceHash, plan[1]!.specs[0]!.sourceHash); + }); + + it('makes one group of two files with one config, and expects the leaves of both', () => { + const more: InstanceSettings = { config: MFA.config, environment: { 'auth_config.reverification': false } }; + const [group, ...rest] = planGroups([file('specs/a.e2e.ts', MFA), file('specs/b.e2e.ts', more)], null); + assert.deepEqual(rest, []); + assert.deepEqual(group!.settings.declared?.environment, { 'user_settings.sign_up.mfa.required': true, 'auth_config.reverification': false }); + assert.equal(group!.settings.key, mfaKey); + }); + + it('refuses two files with one config that expect different values of a leaf, naming both', () => { + const a: InstanceSettings = { config: MFA.config, environment: { 'user_settings.password_settings.min_length': 12 } }; + const b: InstanceSettings = { config: MFA.config, environment: { 'user_settings.password_settings.min_length': 14 } }; + assert.throws(() => planGroups([file('specs/a.e2e.ts', a), file('specs/b.e2e.ts', b)], null), usage(/^specs\/a\.e2e\.ts and specs\/b\.e2e\.ts declare the same config and expect different values of user_settings\.password_settings\.min_length/)); + }); + + it('reports a malformed declaration before anything is planned', () => { + assert.throws(() => planGroups([file('specs/a.e2e.ts'), file('specs/b.e2e.ts', '{ config: {} }')], null), usage(/^specs\/b\.settings\.json: it is not JSON/)); + }); +}); diff --git a/integration/expo-native/test/something-ran.test.ts b/integration/expo-native/test/something-ran.test.ts new file mode 100644 index 00000000000..a7586c823fe --- /dev/null +++ b/integration/expo-native/test/something-ran.test.ts @@ -0,0 +1,47 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { mkdtempSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, it } from 'node:test'; +import { parseE2EReport } from '../src/core/e2e.ts'; +import type { EvidencePath, RunId } from '../src/core/types.ts'; +import { nextStep } from '../src/core/verbs.ts'; + +const run = 'r20261003-000000-abcd' as RunId; +const dir = mkdtempSync(join(tmpdir(), 'verify-ran-')) as EvidencePath; + +function results(rows: readonly { tags?: string[]; status: string; skip?: { cause: string; reason: string } }[]) { + const report = { + schemaVersion: 'report-1', + run: { results: rows.map((row, i) => ({ id: `0000000${i}aa`, kind: 'test', titlePath: [`t${i}`], file: 'specs/golden/auth-start/a.e2e.ts', platform: 'ios', tags: row.tags ?? [], status: row.status, skip: row.skip, attempts: [] })) }, + }; + return parseE2EReport(report, [{ kind: 'golden', path: 'specs/golden/auth-start/a.e2e.ts', feature: null }], dir); +} + +describe('a run in which nothing executed', () => { + it('fails when --grep matched no title', () => { + const grepTypo = results([{ status: 'skipped', skip: { cause: 'filtered', reason: 'title does not match --grep' } }]); + assert.throws(() => nextStep(run, dir, grepTypo, 'auth-start'), { code: 'NO_SPECS', message: /no test ran for auth-start: filtered: title does not match --grep/ }); + }); + + it('fails when the selection registered no tests', () => { + assert.throws(() => nextStep(run, dir, results([]), 'specs/explored/empty.e2e.ts'), { code: 'NO_SPECS', message: /registered no tests/ }); + }); + + it('fails a file whose tests are all test.skip', () => { + const allSkipped = results([{ status: 'skipped', skip: { cause: 'explicit', reason: 'test.skip' } }]); + assert.equal(allSkipped[0]!.status, 'failed', 'an explicitly skipped test is reported as not run, which fails the run'); + assert.doesNotMatch(nextStep(run, dir, allSkipped, 'auth-start'), /attach/); + }); + + it('passes with no attach hint when every selected spec is for the other platform', () => { + const next = nextStep(run, dir, results([{ status: 'skipped', skip: { cause: 'platform-unavailable', reason: 'test declares platforms [android]' } }]), 'sign-in-email-code'); + assert.match(next, /^nothing ran: every selected spec was left out/); + assert.doesNotMatch(next, /attach/); + }); + + it('suggests attach after a run with a passing spec', () => { + assert.match(nextStep(run, dir, results([{ status: 'passed' }]), 'auth-start'), /^\{cli\} attach r20261003-000000-abcd/); + }); +}); diff --git a/integration/expo-native/test/state.test.ts b/integration/expo-native/test/state.test.ts new file mode 100644 index 00000000000..be22cdb0a98 --- /dev/null +++ b/integration/expo-native/test/state.test.ts @@ -0,0 +1,45 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { describe, it } from 'node:test'; +import { Secret } from '../specs/support/secret.ts'; +import { encodeLaunchArguments } from '../specs/support/launch.ts'; +import type { HostLaunch, LaunchId, PublishableKey, RunId, StorageScope } from '../src/core/types.ts'; + +const launch: HostLaunch = { + verifyPublishableKey: 'pk_test_ZXhhbXBsZS5jbGVyay5hY2NvdW50cy5kZXYk' as PublishableKey, + verifyRunId: 'r20261002-141210-7c1e' as RunId, + verifyStorageScope: '0011aabb' as StorageScope, + verifyLaunchId: 'ffee0011' as LaunchId, +}; + +describe('encodeLaunchArguments', () => { + it('encodes iOS launch arguments and Android string extras', () => { + assert.deepEqual(encodeLaunchArguments('ios', launch), [ + '-verifyPublishableKey', launch.verifyPublishableKey, + '-verifyRunId', launch.verifyRunId, + '-verifyStorageScope', '0011aabb', + '-verifyLaunchId', 'ffee0011', + ]); + assert.deepEqual(encodeLaunchArguments('android', launch).slice(0, 3), ['--es', 'verifyPublishableKey', launch.verifyPublishableKey]); + }); + + it('rejects values iOS would parse as a property list', () => { + for (const value of ['(a)', '{a=b}', '', '"quoted"']) { + assert.throws(() => encodeLaunchArguments('ios', { ...launch, verifyInitialIdentifier: value }), /property list/, value); + } + assert.throws(() => encodeLaunchArguments('ios', { ...launch, verifyStorageScope: '{scope}' as StorageScope }), /property list/); + }); + + it('carries an initial identifier to either platform, and nothing for a launch that names none', () => { + const withIdentifier: HostLaunch = { ...launch, verifyInitialIdentifier: '+12015550100' }; + assert.deepEqual(encodeLaunchArguments('ios', withIdentifier).slice(-2), ['-verifyInitialIdentifier', '+12015550100']); + assert.deepEqual(encodeLaunchArguments('android', withIdentifier).slice(-3), ['--es', 'verifyInitialIdentifier', '+12015550100']); + assert.equal(encodeLaunchArguments('android', launch).includes('verifyInitialIdentifier'), false); + assert.throws(() => encodeLaunchArguments('android', { ...launch, verifyInitialIdentifier: '' }), /^Error: launch input verifyInitialIdentifier is empty/); + }); + + it('reads the ticket only through Secret.use', () => { + const args = encodeLaunchArguments('ios', { ...launch, verifySignInTicket: new Secret('ticket', 'tkt_unit_value_123') }); + assert.deepEqual(args.slice(-2), ['-verifySignInTicket', 'tkt_unit_value_123']); + }); +}); diff --git a/integration/expo-native/test/tapping.test.ts b/integration/expo-native/test/tapping.test.ts new file mode 100644 index 00000000000..91f140fcd70 --- /dev/null +++ b/integration/expo-native/test/tapping.test.ts @@ -0,0 +1,86 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { describe, it } from 'node:test'; +import { busyWaits } from '../specs/support/busy-runner.ts'; +import { tapOnceUncovered } from '../specs/support/tapping.ts'; + +const REFUSED = 'perform tap failed: Ref @e5 is covered by another visible element and cannot be tapped safely'; + +function taps(outcomes: readonly (string | null)[]) { + let clock = 0; + let tapped = 0; + return { + count: () => tapped, + tap: async () => { + const outcome = outcomes[Math.min(tapped, outcomes.length - 1)]!; + tapped += 1; + if (outcome !== null) throw new Error(outcome); + }, + settle: (timeoutMs: number) => ({ timeoutMs, now: () => clock, wait: async () => void (clock += 400) }), + }; +} + +describe('a tap on a control that a screen transition still covers', () => { + it('taps again once the transition has uncovered the control', async () => { + const control = taps([REFUSED, REFUSED, null]); + await tapOnceUncovered(control.tap, control.settle(20_000)); + assert.equal(control.count(), 3); + }); + + it('taps once when nothing covers the control', async () => { + const control = taps([null]); + await tapOnceUncovered(control.tap, control.settle(20_000)); + assert.equal(control.count(), 1); + }); + + it('fails with the refusal when the control stays covered past the timeout', async () => { + const control = taps([REFUSED]); + await assert.rejects(tapOnceUncovered(control.tap, control.settle(1_000)), /covered by another visible element/); + assert.equal(control.count(), 4); + }); + + it('does not repeat a tap that failed for another reason', async () => { + const control = taps(['LOCATOR_NOT_FOUND', null]); + await assert.rejects(tapOnceUncovered(control.tap, control.settle(20_000)), /LOCATOR_NOT_FOUND/); + assert.equal(control.count(), 1); + }); +}); + +describe('a tap that the iOS runner refuses because it is still finishing an earlier command', () => { + const BUSY = 'ENGINE_FAILURE: perform tap failed: the iOS automation runner is still finishing a command that overran its watchdog (session e2e-ios-0 on iPhone 17 Pro). The app is fine. Wait a few seconds and rerun.'; + const waiting = (retries: number) => { + const waits: number[] = []; + return { waits, busy: busyWaits(retries, async () => void waits.push(waits.length + 1)) }; + }; + + it('taps again after a wait, and passes when the second tap lands', async () => { + const control = taps([BUSY, null]); + const { waits, busy } = waiting(6); + await tapOnceUncovered(control.tap, control.settle(20_000), busy); + assert.deepEqual([control.count(), waits.length], [2, 1]); + }); + + it('stops at the bound and fails with the refusal', async () => { + const control = taps([BUSY]); + const { waits, busy } = waiting(6); + await assert.rejects(tapOnceUncovered(control.tap, control.settle(20_000), busy), (error: Error) => error.message === BUSY); + assert.deepEqual([control.count(), waits.length], [7, 6]); + }); + + it('fails at once when no wait is allowed, and for any other engine failure', async () => { + const unwaited = taps([BUSY, null]); + await assert.rejects(tapOnceUncovered(unwaited.tap, unwaited.settle(20_000)), /still finishing a command/); + assert.equal(unwaited.count(), 1); + const other = taps(["ENGINE_FAILURE: perform tap failed: the iOS automation runner's main thread overran its watchdog on this command", null]); + const { waits, busy } = waiting(6); + await assert.rejects(tapOnceUncovered(other.tap, other.settle(20_000), busy), /main thread overran its watchdog/); + assert.deepEqual([other.count(), waits.length], [1, 0]); + }); + + it('keeps waiting out a screen transition after the runner is free again', async () => { + const control = taps([BUSY, REFUSED, REFUSED, null]); + const { waits, busy } = waiting(6); + await tapOnceUncovered(control.tap, control.settle(20_000), busy); + assert.deepEqual([control.count(), waits.length], [4, 1]); + }); +}); diff --git a/integration/expo-native/test/test-users.test.ts b/integration/expo-native/test/test-users.test.ts new file mode 100644 index 00000000000..9734526447e --- /dev/null +++ b/integration/expo-native/test/test-users.test.ts @@ -0,0 +1,171 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { mkdtempSync, readFileSync, readdirSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import net from 'node:net'; +import { describe, it } from 'node:test'; +import { inspect } from 'node:util'; +import { CONNECT_ATTEMPT_MS, TEST_PHONES, newTestEmail, parseTestEmail, parseTestPhone, runEmailPrefix, runPassword, testUsers } from '../specs/support/clerk.ts'; +import { newRunId, readClerk } from '../specs/support/inputs.ts'; +import { redact, usedSecretValues } from '../specs/support/secret.ts'; +import { SAMPLE_PUBLISHABLE_KEY, SAMPLE_RUN } from '../testing/sample-inputs.ts'; + +const SECRET_KEY = 'sk_test_testUsersUnitTest0123456789'; + +interface Sent { + readonly method: string; + readonly url: string; + readonly authorization: string | null; + readonly body: Record | null; + readonly signal: AbortSignal | null | undefined; +} + +function clerk(answer: (sent: Sent, n: number) => { readonly status: number; readonly body: unknown }) { + const sent: Sent[] = []; + const fetchImpl = (async (url: string | URL, init?: RequestInit) => { + const request: Sent = { method: init?.method ?? 'GET', url: String(url), authorization: new Headers(init?.headers).get('authorization'), body: typeof init?.body === 'string' ? (JSON.parse(init.body) as Record) : null, signal: init?.signal }; + sent.push(request); + const reply = answer(request, sent.length); + return new Response(JSON.stringify(reply.body), { status: reply.status }); + }) as typeof fetch; + return { sent, fetchImpl }; +} + +const created = (_sent: Sent, n: number) => ({ status: 200, body: { id: `user_${n}`, token: `ticket_${n}_0123456789` } }); + +describe('the users a test asks for', () => { + it('come from api.clerk.com with the secret key when a person gives the keys of a development instance', async () => { + const { sent, fetchImpl } = clerk(created); + const users = testUsers(readClerk({ CLERK_PUBLISHABLE_KEY: SAMPLE_PUBLISHABLE_KEY, CLERK_SECRET_KEY: SECRET_KEY }), SAMPLE_RUN, fetchImpl); + const user = await users.seed(); + await users.signInTicket(user); + assert.deepEqual(sent.map((request) => [request.method, request.url, request.authorization]), [ + ['POST', 'https://api.clerk.com/v1/users', `Bearer ${SECRET_KEY}`], + ['POST', 'https://api.clerk.com/v1/sign_in_tokens', `Bearer ${SECRET_KEY}`], + ]); + assert.deepEqual(sent[1]!.body, { user_id: 'user_1', expires_in_seconds: 120 }); + assert.ok(sent[0]!.signal instanceof AbortSignal, 'each call has a time limit'); + }); + + it('come from the stand-in with its token, and the same requests, when the CLI runs the test', async () => { + const tokenFile = join(mkdtempSync(join(tmpdir(), 'verify-token-')), 'broker-token'); + writeFileSync(tokenFile, 'run-token-0123456789abcdef'); + const { sent, fetchImpl } = clerk(created); + const users = testUsers(readClerk({ CLERK_PUBLISHABLE_KEY: SAMPLE_PUBLISHABLE_KEY, CLERK_E2E_API_URL: 'http://127.0.0.1:4010/v1', CLERK_E2E_API_TOKEN_FILE: tokenFile }), SAMPLE_RUN, fetchImpl); + const user = await users.seed(); + await users.signInTicket(user); + assert.deepEqual(sent.map((request) => [request.method, request.url, request.authorization]), [ + ['POST', 'http://127.0.0.1:4010/v1/users', 'Bearer run-token-0123456789abcdef'], + ['POST', 'http://127.0.0.1:4010/v1/sign_in_tokens', 'Bearer run-token-0123456789abcdef'], + ]); + }); + + const byHand = (answer: Parameters[0]) => { + const fake = clerk(answer); + return { ...fake, users: testUsers(readClerk({ CLERK_PUBLISHABLE_KEY: SAMPLE_PUBLISHABLE_KEY, CLERK_SECRET_KEY: SECRET_KEY }), SAMPLE_RUN, fake.fetchImpl) }; + }; + + it('have an address of this run, a new one each time, and no Clerk call for an address alone', async () => { + const { sent, users } = byHand(created); + const [one, two] = [users.newEmail(), users.newEmail()]; + assert.match(one, /^verify_r20261002_141210_7c1e_[0-9a-f]{8}\+clerk_test@example\.com$/); + assert.notEqual(one, two); + assert.equal(one.startsWith(runEmailPrefix(SAMPLE_RUN)), true); + assert.deepEqual(sent, []); + assert.equal((await users.seed()).email.startsWith(runEmailPrefix(SAMPLE_RUN)), true); + }); + + it('get a password of their own only when asked, and it never shows', async () => { + const { sent, users } = byHand(created); + const plain = await users.seed(); + assert.deepEqual([plain.password, sent[0]!.body!.skip_password_requirement, 'password' in sent[0]!.body!], [null, true, false]); + const one = await users.seed({ password: true }); + const two = await users.seed({ password: true }); + const [first, second] = [String(sent[1]!.body!.password), String(sent[2]!.body!.password)]; + assert.notEqual(first, second, 'each user has its own password'); + for (const typed of [first, second]) assert.match(typed, /^(?=.*[a-z])(?=.*[A-Z])(?=.*\d)(?=.*[^A-Za-z0-9]).{12,}$/, 'it meets the password rules of the standard settings'); + assert.deepEqual([sent[1]!.body!.bypass_client_trust, 'skip_password_requirement' in sent[1]!.body!], [true, false], 'a password sign-in on a new device asks for no second step'); + assert.equal(one.password!.use('device-input', (value) => value), first, 'the test can type the password Clerk was sent'); + for (const shown of [String(one.password), JSON.stringify(one), inspect(one), JSON.stringify(two)]) assert.equal(shown.includes(first) || shown.includes(second), false); + assert.equal(redact(`typed ${first}`), 'typed '); + assert.ok(usedSecretValues().includes(first)); + }); + + it('sign up with the password of the run, which is a secret whole and in each piece a test types it in', () => { + const run = newRunId(); + const password = `Verify-${run}-Pw1!`; + assert.equal(redact(`typed ${password}`), `typed ${password}`, 'nothing knows it before the test process asks for the users of the run'); + testUsers(readClerk({ CLERK_PUBLISHABLE_KEY: SAMPLE_PUBLISHABLE_KEY, CLERK_SECRET_KEY: SECRET_KEY }), run, clerk(created).fetchImpl); + assert.equal(redact(`typed ${password}`), 'typed '); + assert.equal(redact(`input text ${password.slice(0, 16)}`), 'input text '); + assert.equal(redact(`input text ${password.slice(16, 32)}`), 'input text '); + assert.equal(redact(`input text ${password.slice(32)}`), 'input text !', 'one character is not a secret'); + }); + + it('get a password that is a secret in each piece a test types it in', async () => { + const { sent, fetchImpl } = clerk(created); + await testUsers(readClerk({ CLERK_PUBLISHABLE_KEY: SAMPLE_PUBLISHABLE_KEY, CLERK_SECRET_KEY: SECRET_KEY }), SAMPLE_RUN, fetchImpl).seed({ password: true }); + const password = sent[0]!.body!.password as string; + assert.equal(password.length, 28); + assert.equal(redact(`input text ${password.slice(0, 16)} then ${password.slice(16)}`), 'input text then '); + }); + + it('type one text made from the run id in the golden tests, the password the run knows', () => { + const golden = join(import.meta.dirname, '..', 'specs', 'golden'); + const sources = readdirSync(golden, { recursive: true, encoding: 'utf8' }).filter((file) => file.endsWith('.e2e.ts')).map((file) => readFileSync(join(golden, file), 'utf8')); + const filled = sources.flatMap((source) => [...source.matchAll(/host\.fill\([^;]*?(`[^`]*\$\{host\.runId\}[^`]*`)\);/g)].map((match) => match[1])); + assert.deepEqual([...new Set(filled)], ['`Verify-${host.runId}-Pw1!`']); + assert.equal(runPassword(SAMPLE_RUN).use('device-input', (plain) => plain), `Verify-${SAMPLE_RUN}-Pw1!`); + }); + + it('get a test phone no user holds: a taken one moves on to the next, and one test is never handed the same phone twice', async () => { + const free = TEST_PHONES.slice(40, 42); + const held = new Set(TEST_PHONES.filter((phone) => !free.includes(phone))); + const { users } = byHand((sent) => { + const asked = new URL(sent.url).searchParams.get('phone_number'); + if (asked !== null) return { status: 200, body: held.has(asked) ? [{ id: 'user_0' }] : [] }; + const [phone] = (sent.body?.phone_number as string[] | undefined) ?? []; + if (phone !== undefined && held.has(phone)) return { status: 422, body: { errors: [{ code: 'form_identifier_exists' }] } }; + if (phone !== undefined) held.add(phone); + return { status: 200, body: { id: `user_${held.size}` } }; + }); + const reserved = await users.newPhone(); + const seeded = (await users.seed({ phone: true })).phone!; + assert.deepEqual([reserved, seeded].sort(), [...free].sort(), 'the lookup and the seed share out the two numbers nobody holds'); + await assert.rejects(users.newPhone(), /every 555-0100\.\.0199 test phone is taken on this instance/); + await assert.rejects(users.seed({ phone: true }), /every 555-0100\.\.0199 test phone is taken on this instance; delete its test users, or seed without a phone/); + }); + + it('do not go through a hundred phones when the instance takes no phone at all', async () => { + const { sent, users } = byHand(() => ({ status: 422, body: { errors: [{ code: 'form_param_unknown', long_message: 'phone_number is not a valid parameter for this request.' }] } })); + await assert.rejects(users.seed({ phone: true }), /^Error: Clerk POST \/users answered 422: phone_number is not a valid parameter for this request\.$/); + assert.equal(sent.length, 1); + }); + + it("say what Clerk said when it refuses, in Clerk's words", async () => { + const quota = byHand(() => ({ status: 403, body: { errors: [{ code: 'user_quota_exceeded', message: 'Quota exceeded', long_message: 'You have reached your limit of 100 users.' }] } })); + await assert.rejects(quota.users.seed(), /^Error: Clerk POST \/users answered 403: You have reached your limit of 100 users\.$/); + const key = byHand(() => ({ status: 401, body: { errors: [{ code: 'authentication_invalid', message: 'Invalid authentication' }] } })); + await assert.rejects(key.users.signInTicket({ id: 'user_1', email: newTestEmail(SAMPLE_RUN), phone: null, password: null }), /^Error: Clerk POST \/sign_in_tokens answered 401: Invalid authentication$/); + const empty = byHand(() => ({ status: 500, body: null })); + await assert.rejects(empty.users.newPhone(), /^Error: Clerk GET \/users answered 500$/); + }); +}); + +describe('a request to Clerk', () => { + it('gives each address of the host longer to connect than the quarter second Node allows by default', () => { + assert.equal(net.getDefaultAutoSelectFamilyAttemptTimeout(), CONNECT_ATTEMPT_MS); + assert.ok(CONNECT_ATTEMPT_MS >= 2_000); + }); +}); + +describe('test identities', () => { + it('are +clerk_test addresses and 555-01xx phones, and nothing else', () => { + assert.equal(parseTestEmail('verify_1+clerk_test@example.com'), 'verify_1+clerk_test@example.com'); + assert.throws(() => parseTestEmail('someone@example.com'), /is not a \+clerk_test email/); + assert.throws(() => parseTestEmail('verify_1@example.com'), /is not a \+clerk_test email/); + assert.equal(parseTestPhone('+1 (201) 555-0142'), '+12015550142'); + assert.throws(() => parseTestPhone('+1 201 555 0200'), /is not a 555-0100\.\.0199 test phone/); + }); +}); diff --git a/integration/expo-native/test/typing.test.ts b/integration/expo-native/test/typing.test.ts new file mode 100644 index 00000000000..252c3eb0d6b --- /dev/null +++ b/integration/expo-native/test/typing.test.ts @@ -0,0 +1,318 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { describe, it } from 'node:test'; +import { busyWaits } from '../specs/support/busy-runner.ts'; +import { typeConfirmed, type FocusedField } from '../specs/support/typing.ts'; + +function field(reads: readonly (string | null)[], swallows = 0, replaced: () => void = () => {}) { + const calls: string[] = []; + let typed = 0; + let read = 0; + const fake: FocusedField = { + async type(text) { + typed += 1; + calls.push(typed <= swallows ? `type ${text} (swallowed)` : `type ${text}`); + }, + async valueIfReadable() { + calls.push('read'); + const value = reads[Math.min(read, reads.length - 1)]!; + read += 1; + return typed <= swallows ? '' : value; + }, + async valuesOfTheNamedNodes() { + return []; + }, + async refocus() { + calls.push('refocus'); + }, + async tapAgain() { + calls.push('tap again'); + }, + async replace(text) { + calls.push(`replace ${text.length}`); + replaced(); + }, + }; + return { fake, calls }; +} + +function fieldWhoseFirstTapMissed(failures: number, message = 'agent-device type failed: Error (TEXT_INPUT_NOT_FOCUSED): No focused text input was available for typing.') { + const { fake, calls } = field(['Verify']); + let attempts = 0; + const type = fake.type.bind(fake); + const missed: FocusedField = { + ...fake, + async type(text) { + attempts += 1; + if (attempts <= failures) { + calls.push('type refused'); + throw new Error(message); + } + await type(text); + }, + }; + return { fake: missed, calls }; +} + +function fieldWithNoFocusedInput(named: string[], replaced: () => void = () => {}) { + const { fake, calls } = field([null], 0, replaced); + return { fake: { ...fake, valuesOfTheNamedNodes: async () => [...named] }, calls }; +} + +const EMAIL = 'verify_r20261008_122943_f5bd_1+clerk_test@example.com'; +const EMAIL_IN_COMMANDS = ['type verify_r20261008', 'type _122943_f5bd_1+c', 'type lerk_test@exampl', 'type e.com']; + +const settle = { reads: 3, wait: async () => {} }; + +describe('typeConfirmed', () => { + it('types once when the field shows the text', async () => { + const { fake, calls } = field(['Verify']); + await typeConfirmed(fake, 'Verify', settle); + assert.deepEqual(calls, ['type Verify', 'read']); + }); + + it('focuses the field and types once more when the field swallowed the first attempt', async () => { + const { fake, calls } = field(['Verify'], 1); + await typeConfirmed(fake, 'Verify', settle); + assert.deepEqual(calls, ['type Verify (swallowed)', 'read', 'read', 'read', 'refocus', 'type Verify', 'read']); + }); + + it('fails and says the text never reached the field when the second attempt is swallowed too', async () => { + const { fake, calls } = field(['Verify'], 2); + await assert.rejects(typeConfirmed(fake, 'Verify', settle), /the text never reached the field/); + assert.equal(calls.filter((call) => call.startsWith('type')).length, 2); + }); + + it('taps the field again and types when the first tap left nothing focused', async () => { + const { fake, calls } = fieldWhoseFirstTapMissed(1); + await typeConfirmed(fake, 'Verify', settle); + assert.deepEqual(calls, ['type refused', 'tap again', 'type Verify', 'read']); + }); + + it('fails with the typing error when the second tap leaves nothing focused either', async () => { + const { fake, calls } = fieldWhoseFirstTapMissed(2); + await assert.rejects(typeConfirmed(fake, 'Verify', settle), /TEXT_INPUT_NOT_FOCUSED/); + assert.deepEqual(calls, ['type refused', 'tap again', 'type refused']); + }); + + it('taps again on a platform whose typing succeeds into nothing, when it says no field has focus', async () => { + const { fake, calls } = field(['Verify']); + const focus = [false, true]; + const silent: FocusedField = { ...fake, nothingFocused: async () => !focus.shift() }; + await typeConfirmed(silent, 'Verify', settle); + assert.deepEqual(calls, ['tap again', 'type Verify', 'read']); + }); + + it('fails without typing when the second tap leaves no field focused there either', async () => { + const { fake, calls } = field(['Verify']); + const silent: FocusedField = { ...fake, nothingFocused: async () => true }; + await assert.rejects(typeConfirmed(silent, 'Verify', settle), /TEXT_INPUT_NOT_FOCUSED/); + assert.deepEqual(calls, ['tap again']); + }); + + it('does not tap again for any other typing failure', async () => { + const { fake, calls } = fieldWhoseFirstTapMissed(1, 'agent-device type failed: the session is gone'); + await assert.rejects(typeConfirmed(fake, 'Verify', settle), /the session is gone/); + assert.deepEqual(calls, ['type refused']); + }); + + it('sends a long text sixteen characters to a command, so that no typing command outlasts the 30 seconds the iOS runner gives one', async () => { + const { fake, calls } = field([EMAIL]); + await typeConfirmed(fake, EMAIL, settle); + assert.deepEqual(calls, [...EMAIL_IN_COMMANDS, 'read']); + }); + + it('taps again for the first command only, and sends the rest of a long text once', async () => { + const { fake, calls } = fieldWhoseFirstTapMissed(1); + await typeConfirmed({ ...fake, valueIfReadable: async () => EMAIL }, EMAIL, settle); + assert.deepEqual(calls, ['type refused', 'tap again', ...EMAIL_IN_COMMANDS]); + }); + + it('types a swallowed long text again in the same commands', async () => { + const { fake, calls } = field([EMAIL], EMAIL_IN_COMMANDS.length); + await typeConfirmed(fake, EMAIL, settle); + assert.deepEqual(calls.slice(-1 - EMAIL_IN_COMMANDS.length), [...EMAIL_IN_COMMANDS, 'read']); + assert.equal(calls.filter((call) => call === 'refocus').length, 1); + }); + + it('replaces a long text with its first command and types the rest, whatever way the replacing command ended', async () => { + const reads = [EMAIL.slice(1)]; + const { fake, calls } = field(reads, 0, () => { + reads.splice(0, 1, EMAIL); + throw new Error('agent-device fill failed: Error (TEXT_ENTRY_MISMATCH): text entry verification failed'); + }); + await typeConfirmed(fake, EMAIL, settle); + assert.deepEqual(calls.slice(EMAIL_IN_COMMANDS.length + settle.reads), ['replace 16', ...EMAIL_IN_COMMANDS.slice(1), 'read']); + }); + + it('does not type again when the field shows the text a moment after the typing ends', async () => { + const { fake, calls } = field(['', 'Verify']); + await typeConfirmed(fake, 'Verify', settle); + assert.deepEqual(calls, ['type Verify', 'read', 'read']); + }); + + it('types once and confirms nothing when the screen withholds the value, as it does for a secure field', async () => { + const { fake, calls } = field([null]); + await typeConfirmed(fake, 'a password', settle); + assert.deepEqual(calls, ['type a password', 'read']); + }); + + it('confirms nothing for a secure field that reads as bullets', async () => { + const { fake, calls } = field(['\u2022'.repeat(9)]); + await typeConfirmed(fake, 'Verify-Pw1!', settle); + assert.deepEqual(calls, ['type Verify-Pw1!', 'read']); + }); + + it('accepts the formatting a field adds and the case it changes', async () => { + for (const [typed, shown] of [['2015550124', '+1 (201) 555-0124'], ['Ada+clerk_test@Example.com', 'ada+clerk_test@example.com'], ['424242', '4 2 4 2 4 2']] as const) { + const { fake, calls } = field([shown]); + await typeConfirmed(fake, typed, settle); + assert.deepEqual(calls.filter((call) => !call.startsWith('type ')), ['read'], shown); + } + }); + + it('replaces the contents once when a character was dropped, and passes when the field then holds the text', async () => { + const reads = ['44242']; + const { fake, calls } = field(reads, 0, () => reads.splice(0, 1, '424242')); + await typeConfirmed(fake, '424242', settle); + assert.deepEqual(calls, ['type 424242', 'read', 'read', 'read', 'replace 6', 'read']); + }); + + it('judges a replacement by what the field then holds, not by how the command that made it ended', async () => { + const reads: (string | null)[] = ['(215) 550-198']; + const formatted = field(reads, 0, () => { + reads.splice(0, 1, '(201) 555-0198'); + throw new Error('agent-device fill failed: Error (TEXT_ENTRY_MISMATCH): text entry verification failed'); + }); + await typeConfirmed(formatted.fake, '2015550198', settle); + assert.equal(formatted.calls.at(-1), 'read'); + + const code: (string | null)[] = ['44242']; + const submitted = field(code, 0, () => { + code.splice(0, 1, null); + throw new Error('agent-device fill failed: the screen changed while the code was typed'); + }); + await typeConfirmed(submitted.fake, '424242', settle); + assert.equal(submitted.calls.at(-1), 'read'); + + const stillWrong = field(['44242'], 0, () => { + throw new Error('agent-device fill failed: the session is gone'); + }); + await assert.rejects(typeConfirmed(stillWrong.fake, '424242', settle), /6 letters and digits were typed, and after one attempt to replace its contents the field holds 5$/); + }); + + it('fails with the two lengths and neither value when the replacement leaves the field wrong too', async () => { + const { fake, calls } = field(['Verify-Pw']); + const failure = await typeConfirmed(fake, 'Verify-Pw1!', settle).then(() => null, (error: Error) => error.message); + assert.equal(failure, 'the field does not hold the typed text: 9 letters and digits were typed, and after one attempt to replace its contents the field holds 8'); + assert.equal(calls.filter((call) => call.startsWith('replace')).length, 1); + assert.equal(calls.filter((call) => call.startsWith('type')).length, 1); + }); + + it('reads the nodes the test named when no focused text input can be read, and replaces a wrong value there', async () => { + const labelAndInput = ['', '442422']; + const { fake, calls } = fieldWithNoFocusedInput(labelAndInput, () => labelAndInput.splice(1, 1, '424242')); + await typeConfirmed(fake, '424242', settle); + assert.deepEqual(calls.filter((call) => call !== 'read'), ['type 424242', 'replace 6']); + }); + + it('fails when the named node holds the digits in another order after the replacement too', async () => { + const { fake } = fieldWithNoFocusedInput(['442422']); + await assert.rejects(typeConfirmed(fake, '424242', settle), /6 letters and digits were typed, and after one attempt to replace its contents the field holds 6$/); + }); + + it('makes no claim when no named node shows a value, as a node that is not a text input shows none, or when two show different values', async () => { + for (const named of [[''], [], ['442422', '424242']]) { + const { fake, calls } = fieldWithNoFocusedInput(named); + await typeConfirmed(fake, '424242', settle); + assert.deepEqual(calls, ['type 424242', 'read'], JSON.stringify(named)); + } + }); + + it('does not replace when the screen moved on before the value could be read, as it does when a code field submits itself', async () => { + const { fake, calls } = field(['4242', null]); + await typeConfirmed(fake, '424242', settle); + assert.deepEqual(calls, ['type 424242', 'read', 'read']); + }); + + describe('when the iOS runner refuses a typing command because it is still finishing an earlier one', () => { + const BUSY = 'agent-device type failed: Error (COMMAND_FAILED): The iOS runner is still finishing a previous command that exceeded its execution watchdog (usually an accessibility capture on a heavy or animating screen).'; + + function busyField(options: { readonly reads: (string | null)[]; readonly refuse: (call: 'type' | 'replace', nth: number) => boolean; readonly onReplace?: () => void }) { + const calls: string[] = []; + let waits = 0; + let sent = 0; + const refusedOr = (call: 'type' | 'replace', text: string, land: () => void): void => { + sent += 1; + if (options.refuse(call, sent)) { + calls.push(`${call} refused`); + throw new Error(`${BUSY} (command ${sent})`); + } + calls.push(call === 'type' ? `type ${text}` : `replace ${text.length}`); + land(); + }; + const fake: FocusedField = { + type: async (text) => refusedOr('type', text, () => {}), + replace: async (text) => refusedOr('replace', text, options.onReplace ?? (() => {})), + valueIfReadable: async () => (calls.push('read'), options.reads[0]!), + valuesOfTheNamedNodes: async () => [], + refocus: async () => void calls.push('refocus'), + tapAgain: async () => void calls.push('tap again'), + }; + return { fake, calls, waits: () => waits, busy: (most: number) => busyWaits(most, async () => void ((waits += 1), calls.push('wait'))) }; + } + + it('waits, then replaces the contents instead of typing the command again, so characters that landed are not typed twice', async () => { + const reads = ['Ver']; + const refused = busyField({ reads, refuse: (_call, nth) => nth === 1, onReplace: () => reads.splice(0, 1, 'Verify') }); + await typeConfirmed(refused.fake, 'Verify', settle, refused.busy(6)); + assert.deepEqual(refused.calls, ['type refused', 'wait', 'replace 6', 'read']); + }); + + it('starts the replacement of a long text again from its first command when a later command is refused', async () => { + const refused = busyField({ reads: [EMAIL], refuse: (_call, nth) => nth === 2 || nth === 4 }); + await typeConfirmed(refused.fake, EMAIL, settle, refused.busy(6)); + assert.deepEqual(refused.calls, ['type verify_r20261008', 'type refused', 'wait', 'replace 16', 'type refused', 'wait', 'replace 16', ...EMAIL_IN_COMMANDS.slice(1), 'read']); + }); + + it('stops at the bound and fails with the first refusal', async () => { + const refused = busyField({ reads: ['Verify'], refuse: () => true }); + await assert.rejects(typeConfirmed(refused.fake, 'Verify', settle, refused.busy(3)), (error: Error) => error.message === `${BUSY} (command 1)`); + assert.deepEqual(refused.calls, ['type refused', 'wait', 'replace refused', 'wait', 'replace refused', 'wait', 'replace refused']); + assert.equal(refused.waits(), 3); + }); + + it('has one bound for the whole fill, so the waits of its typing count against the replacement of a wrong value', async () => { + const refused = busyField({ reads: ['Verfy'], refuse: (_call, nth) => nth !== 3 }); + await assert.rejects(typeConfirmed(refused.fake, 'Verify', settle, refused.busy(3)), (error: Error) => error.message === `${BUSY} (command 5)`); + assert.deepEqual(refused.calls, ['type refused', 'wait', 'replace refused', 'wait', 'replace 6', 'read', 'read', 'read', 'replace refused', 'wait', 'replace refused']); + assert.equal(refused.waits(), 3); + }); + + it('still fails at the fill when the replacement leaves the field wrong', async () => { + const refused = busyField({ reads: ['Verfy'], refuse: (_call, nth) => nth === 1 }); + await assert.rejects(typeConfirmed(refused.fake, 'Verify', settle, refused.busy(6)), /after one attempt to replace its contents the field holds 5$/); + assert.equal(refused.calls.filter((call) => call === 'type Verify').length, 0, 'the refused command is never typed again'); + }); + + it('waits and replaces again when the runner refuses the replacement of a wrong value', async () => { + const reads = ['44242']; + const refused = busyField({ reads, refuse: (call, nth) => call === 'replace' && nth === 2, onReplace: () => reads.splice(0, 1, '424242') }); + await typeConfirmed(refused.fake, '424242', settle, refused.busy(6)); + assert.deepEqual(refused.calls, ['type 424242', 'read', 'read', 'read', 'replace refused', 'wait', 'replace 6', 'read']); + }); + + it('fails with the refusal at once when no wait is allowed, as before', async () => { + const refused = busyField({ reads: ['Verify'], refuse: (_call, nth) => nth === 1 }); + await assert.rejects(typeConfirmed(refused.fake, 'Verify', settle), (error: Error) => error.message === `${BUSY} (command 1)`); + assert.deepEqual(refused.calls, ['type refused']); + }); + + it('does not wait or replace for any other typing failure', async () => { + const { fake, calls } = fieldWhoseFirstTapMissed(1, 'agent-device type failed: Error (COMMAND_FAILED): main thread execution timed out'); + const waited: string[] = []; + await assert.rejects(typeConfirmed(fake, 'Verify', settle, busyWaits(6, async () => void waited.push('wait'))), /main thread execution timed out/); + assert.deepEqual([calls, waited], [['type refused'], []]); + }); + }); +}); diff --git a/integration/expo-native/test/waiting.test.ts b/integration/expo-native/test/waiting.test.ts new file mode 100644 index 00000000000..f915e819375 --- /dev/null +++ b/integration/expo-native/test/waiting.test.ts @@ -0,0 +1,324 @@ +import '../testing/git-env.ts'; +import assert from 'node:assert/strict'; +import { describe, it } from 'node:test'; +import { newTestEmail } from '../specs/support/clerk.ts'; +import { SAVE_PASSWORD_PROMPT, errorScreenElseSavePasswordPrompt, onScreen, signedInAs, signedOut, until, type Home, type ScreenElement, type Sight, type WaitPace, type WaitScreen } from '../specs/support/waiting.ts'; +import type { SeededUser } from '../src/core/types.ts'; +import { SAMPLE_RUN } from '../testing/sample-inputs.ts'; + +const EMAIL = newTestEmail(SAMPLE_RUN); +const HEADING = `Signed in as ${EMAIL}`; +const SEEDED: SeededUser = { id: 'user_seeded', email: EMAIL, phone: null, password: null }; +const WAITING_FOR = 'the signed-in home'; +const POLL_MS = 400; +const SNAPSHOT_FAILED = 'ENGINE_FAILURE: snapshot failed'; + +type Texts = string | readonly string[] | null; + +interface Showing { + heading: Texts; + userId: Texts; + sessionId: Texts; + signedOut: boolean; + signIn: boolean; + prompt: boolean; + errorScreen: string | null; +} + +const ELSEWHERE: Showing = { heading: null, userId: null, sessionId: null, signedOut: false, signIn: false, prompt: false, errorScreen: null }; +const SIGNED_IN: Showing = { ...ELSEWHERE, heading: HEADING, userId: 'user_1', sessionId: 'sess_1' }; +const SIGNED_OUT: Showing = { ...ELSEWHERE, signedOut: true, signIn: true }; + +interface Simulated { + readonly readMs?: (read: number) => number; + readonly dismissMs?: number; + readonly dismissal?: 'throws'; + readonly homeReadableUnderThePrompt?: true; + readonly unreadable?: readonly string[]; + readonly afterRead?: (read: number, show: (next: Partial) => void) => void; +} + +function device(start: Showing, simulated: Simulated = {}) { + let showing = { ...start }; + let clock = 0; + let dismissals = 0; + let waits = 0; + const reads: string[] = []; + const readsPerPass: number[] = [0]; + const show = (next: Partial): void => void (showing = { ...showing, ...next }); + const element = (what: string, texts: () => readonly string[]): ScreenElement => ({ + async allTextContents() { + reads.push(what); + readsPerPass[readsPerPass.length - 1]! += 1; + clock += simulated.readMs?.(reads.length) ?? 100; + const shown = simulated.unreadable?.includes(what) === true ? null : texts(); + simulated.afterRead?.(reads.length, show); + if (shown === null) throw new Error(SNAPSHOT_FAILED); + return shown; + }, + }); + const behindThePrompt = (texts: () => Texts) => (): readonly string[] => { + const now = texts(); + return (showing.prompt && simulated.homeReadableUnderThePrompt !== true) || now === null ? [] : typeof now === 'string' ? [now] : now; + }; + const home: Home = { + signedIn: element('heading', behindThePrompt(() => showing.heading)), + userId: element('user ID', behindThePrompt(() => showing.userId)), + sessionId: element('session ID', behindThePrompt(() => showing.sessionId)), + signedOut: element('signed out', behindThePrompt(() => (showing.signedOut ? 'Signed out' : null))), + }; + const screen: WaitScreen = { + home, + errorScreenElsePrompt: element('in the way', () => (showing.errorScreen !== null ? [showing.errorScreen] : showing.prompt ? [SAVE_PASSWORD_PROMPT, SAVE_PASSWORD_PROMPT] : [])), + dismissPrompt: () => { + dismissals += 1; + clock += simulated.dismissMs ?? 700; + if (simulated.dismissal === 'throws') throw new Error('LOCATOR_NOT_FOUND: Not Now'); + show({ prompt: false }); + return Promise.resolve(); + }, + }; + const pace = (timeoutMs: number): WaitPace => ({ + timeoutMs, + now: () => clock, + wait: async () => { + waits += 1; + clock += POLL_MS; + readsPerPass.push(0); + }, + }); + return { + home, + signIn: element('sign-in button', behindThePrompt(() => (showing.signIn ? 'Sign in' : null))), + reads, + readsPerPass, + clock: () => clock, + dismissals: () => dismissals, + waits: () => waits, + wait: (sight: Sight, timeoutMs = 30_000): Promise => until(sight, WAITING_FOR, screen, pace(timeoutMs)), + }; +} + +const failure = async (wait: Promise): Promise => { + const error = await wait.then( + () => null, + (thrown: unknown) => thrown, + ); + assert.ok(error instanceof Error, 'the wait passed'); + return error.message; +}; + +describe('a wait for the home to show the user signed in', () => { + it('passes when the home is signed in only once the save-password prompt is dismissed, and the tap that dismisses it ends after the deadline', async () => { + const phone = device({ ...SIGNED_IN, prompt: true }, { readMs: (read) => (read === 2 ? 27_000 : 800), dismissMs: 2_800 }); + await phone.wait(signedInAs(phone.home, EMAIL)); + assert.equal(phone.dismissals(), 1); + assert.ok(phone.clock() > 30_000, `the home was read at ${phone.clock()}ms, which is not after the deadline`); + }); + + it('reads the home once more after the deadline before it fails, so a home that signed in during the last slow read passes, and it does not sleep before that reading', async () => { + const phone = device(ELSEWHERE, { + readMs: (read) => (read === 2 ? 31_000 : 100), + afterRead: (read, show) => { + if (read === 2) show(SIGNED_IN); + }, + }); + await phone.wait(signedInAs(phone.home, EMAIL)); + assert.deepEqual(phone.reads, ['heading', 'in the way', 'heading', 'user ID', 'session ID', 'signed out']); + assert.equal(phone.waits(), 0); + }); + + it('dismisses a prompt it first meets in its last pass, and lets the reading after that decide', async () => { + const phone = device(ELSEWHERE, { + readMs: (read) => (read === 1 ? 30_000 : 100), + afterRead: (read, show) => { + if (read === 2) show({ ...SIGNED_IN, prompt: true }); + }, + }); + await phone.wait(signedInAs(phone.home, EMAIL)); + assert.equal(phone.dismissals(), 1); + }); + + it('fails with what the deciding reading saw, and reads nothing for the message after it has looked for the error screen and the prompt', async () => { + const LAST_READ_OF_THE_HOME = 8; + const phone = device(SIGNED_OUT, { + afterRead: (read, show) => { + if (read === LAST_READ_OF_THE_HOME) show(SIGNED_IN); + }, + }); + const message = await failure(phone.wait(signedInAs(phone.home, EMAIL), 1_000)); + assert.ok(message.startsWith('the app did not show the signed-in home within 1000ms; the home shows "Signed out"; passes: 3; '), message); + assert.doesNotMatch(message, /Signed in as/); + assert.deepEqual(phone.reads.slice(4), ['heading', 'user ID', 'session ID', 'signed out', 'in the way']); + }); + + it('makes two captures in a pass that waits, one for the heading and one for the error screen and the prompt together, and three in the pass that finds the user signed in', async () => { + const phone = device(ELSEWHERE, { + afterRead: (read, show) => { + if (read === 4) show(SIGNED_IN); + }, + }); + await phone.wait(signedInAs(phone.home, EMAIL)); + assert.deepEqual(phone.readsPerPass, [2, 2, 3]); + assert.deepEqual(phone.reads, ['heading', 'in the way', 'heading', 'in the way', 'heading', 'user ID', 'session ID']); + }); + + it('makes five captures in a pass that dismisses the prompt over the home: two that find it and three that find the user signed in behind it', async () => { + const phone = device({ ...SIGNED_IN, prompt: true }); + await phone.wait(signedInAs(phone.home, EMAIL)); + assert.deepEqual(phone.readsPerPass, [5]); + }); + + it('makes no more than seven captures in a pass before the deadline: three for the home, one for what is in the way, and three for the home again after a prompt', async () => { + const phone = device( + { ...SIGNED_IN, sessionId: null, prompt: true }, + { + homeReadableUnderThePrompt: true, + afterRead: (read, show) => { + if (read === 7) show({ sessionId: 'sess_1' }); + }, + }, + ); + await phone.wait(signedInAs(phone.home, EMAIL)); + assert.deepEqual(phone.readsPerPass, [7, 3]); + assert.deepEqual(phone.reads.slice(0, 7), ['heading', 'user ID', 'session ID', 'in the way', 'heading', 'user ID', 'session ID']); + }); + + it('reads each of the four elements of the home once in its last pass, and judges and describes from those four', async () => { + const phone = device(SIGNED_OUT); + await failure(phone.wait(signedInAs(phone.home, EMAIL), 0)); + assert.deepEqual(phone.reads, ['heading', 'user ID', 'session ID', 'signed out', 'in the way']); + }); + + it('reports a read that threw with its error, and does not call the heading missing', async () => { + const phone = device(SIGNED_IN, { unreadable: ['heading'] }); + const message = await failure(phone.wait(signedInAs(phone.home, EMAIL), 1_000)); + assert.ok(message.includes(`; the home shows user ID user_1, session ID sess_1; unread: the heading (${SNAPSHOT_FAILED}); passes: 3; `), message); + assert.ok(message.endsWith(`; failed reads: 3, the last: the heading: ${SNAPSHOT_FAILED}`), message); + }); + + it('says that the home could not be read when no read of it succeeded, and does not call it missing', async () => { + const phone = device(SIGNED_IN, { unreadable: ['heading', 'user ID', 'session ID', 'signed out'] }); + const message = await failure(phone.wait(signedInAs(phone.home, EMAIL), 0)); + assert.ok(message.includes(`within 0ms; the home could not be read; unread: the heading (${SNAPSHOT_FAILED}), the user ID (${SNAPSHOT_FAILED}), the session ID (${SNAPSHOT_FAILED}), "Signed out" (${SNAPSHOT_FAILED}); passes: 1; `), message); + assert.doesNotMatch(message, /not on screen/); + }); + + it('goes on when it cannot read what is in the way, and reports that read too', async () => { + const phone = device(ELSEWHERE, { unreadable: ['in the way'] }); + const message = await failure(phone.wait(signedInAs(phone.home, EMAIL), 1_000)); + assert.ok(message.endsWith(`; failed reads: 3, the last: the error screen or the save-password prompt: ${SNAPSHOT_FAILED}`), message); + assert.equal(phone.dismissals(), 0); + }); + + it('goes on to its deadline when the tap that should dismiss the prompt fails', async () => { + const phone = device({ ...SIGNED_IN, prompt: true }, { dismissal: 'throws' }); + const message = await failure(phone.wait(signedInAs(phone.home, EMAIL), 1_000)); + assert.ok(message.startsWith('the app did not show the signed-in home within 1000ms; the home is not on screen; passes: '), message); + assert.ok(phone.dismissals() > 1); + }); + + it('says how many passes ran, how many reads they made, and how long the slowest read took', async () => { + const phone = device(ELSEWHERE, { readMs: (read) => (read === 4 ? 1_500 : 100) }); + const message = await failure(phone.wait(signedInAs(phone.home, EMAIL), 1_000)); + assert.ok(message.endsWith('; the home is not on screen; passes: 3; reads: 9, the slowest 1500ms (the error screen or the save-password prompt)'), message); + }); + + it('fails at once with the reason when the app shows its error screen, taps nothing, and still says what the reads did', async () => { + const phone = device({ ...ELSEWHERE, errorScreen: 'Something went wrong The publishable key is missing' }); + const message = await failure(phone.wait(signedInAs(phone.home, EMAIL))); + assert.equal(message, 'the app did not show the signed-in home; it shows its error screen: Something went wrong The publishable key is missing; passes: 1; reads: 2, the slowest 100ms (the heading)'); + assert.equal(phone.dismissals(), 0); + }); + + const passes = async (showing: Showing, who: SeededUser | typeof EMAIL): Promise => { + const phone = device(showing); + return phone.wait(signedInAs(phone.home, who), 1_000).then( + () => true, + () => false, + ); + }; + it('passes on the heading, a user ID and a session ID after a sign-up', async () => assert.equal(await passes(SIGNED_IN, EMAIL), true)); + it('does not pass on the heading of another address', async () => assert.equal(await passes({ ...SIGNED_IN, heading: 'Signed in as someone-else@example.com' }, EMAIL), false)); + it('does not pass on the heading on two nodes', async () => assert.equal(await passes({ ...SIGNED_IN, heading: [HEADING, HEADING] }, EMAIL), false)); + it('does not pass on the heading and a session ID with no user ID', async () => assert.equal(await passes({ ...SIGNED_IN, userId: null }, EMAIL), false)); + it('does not pass on the heading and a user ID with no session ID', async () => assert.equal(await passes({ ...SIGNED_IN, sessionId: null }, EMAIL), false)); + it('does not pass on an empty session ID', async () => assert.equal(await passes({ ...SIGNED_IN, sessionId: '' }, EMAIL), false)); + it('does not pass on two session IDs', async () => assert.equal(await passes({ ...SIGNED_IN, sessionId: ['sess_1', 'sess_2'] }, EMAIL), false)); + it('passes on the ID of the seeded user', async () => assert.equal(await passes({ ...SIGNED_IN, userId: SEEDED.id }, SEEDED), true)); + it('does not pass on the ID of another user than the seeded one', async () => assert.equal(await passes(SIGNED_IN, SEEDED), false)); + + it('names every value the home shows when it fails on one of them', async () => { + const phone = device(SIGNED_IN); + const message = await failure(phone.wait(signedInAs(phone.home, SEEDED), 1_000)); + assert.ok(message.includes(`; the home shows "${HEADING}", user ID user_1, session ID sess_1; passes: `), message); + }); +}); + +describe('a wait for the home to show the user signed out', () => { + it('passes on "Signed out" with no heading, user ID or session ID, in four captures', async () => { + const phone = device(SIGNED_OUT); + await phone.wait(signedOut(phone.home)); + assert.deepEqual(phone.reads, ['signed out', 'heading', 'user ID', 'session ID']); + }); + + const stillShows = async (left: Partial): Promise => { + const phone = device({ ...SIGNED_OUT, ...left }); + return failure(phone.wait(signedOut(phone.home), 1_000)); + }; + it('does not pass while the heading of a signed-in user is still on the home', async () => assert.match(await stillShows({ heading: 'left over' }), /within 1000ms/)); + it('does not pass while the userId of a signed-in user is still on the home', async () => assert.match(await stillShows({ userId: 'left over' }), /within 1000ms/)); + it('does not pass while the sessionId of a signed-in user is still on the home', async () => assert.match(await stillShows({ sessionId: 'left over' }), /within 1000ms/)); + + it('names the user ID that is still on the home beside "Signed out"', async () => { + const phone = device({ ...SIGNED_OUT, userId: 'user_1' }); + const message = await failure(phone.wait(signedOut(phone.home), 1_000)); + assert.ok(message.includes('; the home shows "Signed out", user ID user_1, session ID none; passes: '), message); + }); + + const unread = async (element: string): Promise => { + const phone = device(SIGNED_OUT, { unreadable: [element] }); + return failure(phone.wait(signedOut(phone.home), 1_000)); + }; + const saysUnread = (named: string): string => `; the home shows "Signed out"; unread: ${named} (${SNAPSHOT_FAILED}); passes: `; + it('does not take a heading it could not read for one that is gone', async () => assert.ok((await unread('heading')).includes(saysUnread('the heading')))); + it('does not take a user ID it could not read for one that is gone', async () => assert.ok((await unread('user ID')).includes(saysUnread('the user ID')))); + it('does not take a session ID it could not read for one that is gone', async () => assert.ok((await unread('session ID')).includes(saysUnread('the session ID')))); +}); + +describe('a wait for a launch to land on a control', () => { + it('passes once the control is on screen, in one capture', async () => { + const phone = device(SIGNED_OUT); + await phone.wait(onScreen(phone.signIn, 'the "Sign in" button')); + assert.deepEqual(phone.reads, ['sign-in button']); + }); + + it('fails with what the home shows in its place', async () => { + const phone = device(SIGNED_IN); + const message = await failure(phone.wait(onScreen(phone.signIn, 'the "Sign in" button'), 1_000)); + assert.ok(message.includes(`within 1000ms; the home shows "${HEADING}", user ID user_1, session ID sess_1; passes: `), message); + }); + + it('reports the error of a control it could not read', async () => { + const phone = device(SIGNED_OUT, { unreadable: ['sign-in button'] }); + const message = await failure(phone.wait(onScreen(phone.signIn, 'the "Sign in" button'), 1_000)); + assert.ok(message.includes(`; the home shows "Signed out"; unread: the "Sign in" button (${SNAPSHOT_FAILED}); passes: `), message); + }); + + it('reads a control that is part of the home once in its last pass, so the message cannot show a heading the verdict did not see', async () => { + const phone = device(ELSEWHERE, { + afterRead: (read, show) => { + if (read === 1) show(SIGNED_IN); + }, + }); + const message = await failure(phone.wait(onScreen(phone.home.signedIn, 'the heading'), 0)); + assert.ok(message.includes('within 0ms; the home shows user ID user_1, session ID sess_1; passes: 1; '), message); + assert.deepEqual(phone.reads, ['heading', 'user ID', 'session ID', 'signed out', 'in the way']); + }); +}); + +describe('the selector that reads the error screen and the save-password prompt in one capture', () => { + it('names the error screen first, so the prompt is read only when the app shows no error screen', () => { + assert.equal(errorScreenElseSavePasswordPrompt('e2e.launch.error'), 'id="e2e.launch.error" || label="Save Password?"'); + }); +}); diff --git a/integration/expo-native/testing/sample-inputs.ts b/integration/expo-native/testing/sample-inputs.ts new file mode 100644 index 00000000000..fbed7b3ade9 --- /dev/null +++ b/integration/expo-native/testing/sample-inputs.ts @@ -0,0 +1,19 @@ +import type { Inputs, TestApp } from '../specs/support/inputs.ts'; +import type { PublishableKey, RunId } from '../specs/support/types.ts'; + +export const SAMPLE_RUN = 'r20261002-141210-7c1e' as RunId; +export const SAMPLE_PUBLISHABLE_KEY = 'pk_test_ZXhhbXBsZS5jbGVyay5hY2NvdW50cy5kZXYk' as PublishableKey; + +export const IOS_APP: TestApp = { platforms: ['ios'], id: () => 'com.clerk.sample', entry: () => ({ kind: 'binary' }) }; + +export const BOTH_PLATFORMS_APP: TestApp = { + platforms: ['ios', 'android'], + id: (platform) => (platform === 'ios' ? 'com.clerk.sample' : 'com.clerk.sample.android'), + entry: (_platform, devServer) => (devServer === null ? { kind: 'binary' } : { kind: 'dev-client', launchArguments: ['--initialUrl', devServer], openLink: null, androidActivity: '.MainActivity' }), +}; + +export const SAMPLE_INPUTS: Inputs = { + target: { platform: 'ios', device: { kind: 'local', id: 'FDF0CD9E-CF9E-42B6-AE3A-116A665F7EF3' }, session: 'verify-ios-abc', build: { path: null, devServer: null } }, + clerk: { publishableKey: SAMPLE_PUBLISHABLE_KEY, access: { kind: 'stand-in', url: 'http://127.0.0.1:4010/v1', tokenFile: '/package/.verify/scratch/r/broker-token' } }, + run: SAMPLE_RUN, +}; From df8ab4686e723af492ab0ff7b420b834035f7359 Mon Sep 17 00:00:00 2001 From: Mike Pitre <12040919+mikepitre@users.noreply.github.com> Date: Wed, 7 Oct 2026 20:31:02 -0400 Subject: [PATCH 7/7] docs(expo): document the skill for agents and contributors Co-Authored-By: Claude Opus 5.5 --- .claude/skills/verify-clerk-expo/SKILL.md | 217 ++++++++++++++++++ .../verify-clerk-expo/references/devices.md | 10 + .../verify-clerk-expo/references/freshness.md | 39 ++++ .../verify-clerk-expo/references/instances.md | 81 +++++++ AGENTS.md | 5 + packages/expo/AGENTS.md | 6 + 6 files changed, 358 insertions(+) create mode 100644 .claude/skills/verify-clerk-expo/SKILL.md create mode 100644 .claude/skills/verify-clerk-expo/references/devices.md create mode 100644 .claude/skills/verify-clerk-expo/references/freshness.md create mode 100644 .claude/skills/verify-clerk-expo/references/instances.md diff --git a/.claude/skills/verify-clerk-expo/SKILL.md b/.claude/skills/verify-clerk-expo/SKILL.md new file mode 100644 index 00000000000..24b25e38e8f --- /dev/null +++ b/.claude/skills/verify-clerk-expo/SKILL.md @@ -0,0 +1,217 @@ +--- +name: verify-clerk-expo +description: Drive @clerk/expo in the expo-native test app (native AuthView, UserButton, UserProfileView, custom useSignIn and useSignUp flows, token cache, the Google and biometrics native modules) on an iOS simulator or Android emulator against a real Clerk development instance that the session creates and deletes, and capture video, screenshots, and the app log as evidence. Use it to prove any change to packages/expo or the test app works before calling it done, to reproduce a UI bug, or to run the golden regression specs. +--- + +# verify-clerk-expo + +`integration/expo-native/bin/control-clerk-expo` is a control CLI over [e2e](https://github.com/tester-army/e2e) 0.18.0 and `@e2e-dev/mobile` 0.10.0. It builds the `expo-native` test app in `integration/templates/expo-native`, leases a simulator or emulator, creates one Clerk application for the worktree, seeds `+clerk_test` users, runs specs, and keeps the evidence. The test app is a Debug dev client, and Metro serves your working tree to it. + +The tests are an ordinary e2e project. `e2e.config.ts` and `specs/` run under `npx e2e run` when the environment names a device, a build of the test app, and a development instance's keys. The CLI sits on top: it makes those three for a run and keeps the evidence. [The package README](../../../integration/expo-native/README.md) has the commands for a run by hand and what such a run leaves out. + +No change to `@clerk/expo` UI or auth behavior is done until a `run` on the real test app shows the changed behavior, on each platform the change touches. + +Run every command from the repo root. In the prose below, `doctor`, `up`, `run`, `screen`, `attach`, and `down` are verbs of that CLI. The tests and the CLI are the Node package `integration/expo-native/`, and paths that begin `specs/`, `src/`, `test/`, or `.verify/` are inside it. `features/` and `references/` are beside this file. Every verb but `attach` takes `--platform ios|android`, and iOS is the default. Every verb takes `--json` and then prints one `{ "ok": ... }` object. Exit codes are 0 for success, 1 for a failing spec, 2 for a usage error, and 3 for a failed precondition. Every error prints a `fix` line. + +CI runs none of these specs. A regression test that must run on every pull request belongs in `integration/tests/expo-native/`, which `.github/workflows/expo-native-build.yml` runs against a Release build of the same test app with the same `e2e` engine. + +The test app links `@clerk/expo`, `@clerk/expo-biometrics`, and `@clerk/expo-google-signin` from the workspace. It does not install `@clerk/expo-passkeys`, so the skill cannot verify passkeys. + +## Launch + +Set up each machine once. + +1. Install Node 24, at 24.8.0 or newer. For a local iOS device, install Xcode with an iOS simulator runtime. For a local Android device, install Android Studio with the SDK, the emulator, and Java 21. The CLI looks for the SDK in `ANDROID_HOME`, `ANDROID_SDK_ROOT`, and `~/Library/Android/sdk`. +2. Create the iOS template simulator, which the CLI clones to make each simulator it drives, for example with `xcrun simctl clone "iPhone Air" "Clerk Verify Template iOS"`. Android needs no template: the first `up --platform android` writes the `Clerk_Verify_Pixel` AVD. +3. Give the machine the team's Clerk Platform API key. Set `CLERK_PLATFORM_API_KEY` to the key. Or set `CLERK_PLATFORM_API_KEY_FILE` to a file that holds it and that only you can read (mode 0600). Or set `VERIFY_PLATFORM_KEY_REFERENCE` to the key's 1Password secret reference, which the 1Password CLI reads once a person approves the request in the 1Password app, within 60 seconds. Never put the key or the reference in a file inside a repository. + +Then, in each worktree: + +```console +$ pnpm install # once per worktree +$ npm ci --prefix integration/expo-native # once per worktree +$ integration/expo-native/bin/control-clerk-expo doctor --platform ios +$ integration/expo-native/bin/control-clerk-expo up --platform ios +backend local this Mac runs the simulator itself +instance creating verify-throwaway-until-- in org_3KHungJxbvIscuSvy8oos5MHAli +build local building... +build turbo build @clerk/expo, @clerk/expo-biometrics, @clerk/expo-google-signin +instance up in 0.7s on standard, 62 settings match src/core/instances/base.json +build expo prebuild --clean --platform ios +build xcodebuild Debug (dev client) +build local built in 110s +device verify-ios-1 cloning Clerk Verify Template iOS +install on verify-ios-1 +watch packages/expo tsdown --watch (pid ) +metro :8082 expo start (pid ) +metro :8082 bundling ios once so the first launch does not wait on Metro +device verify-ios-1 local leased by this worktree installed +``` + +The package is outside the pnpm workspace, so `npm ci` installs its pinned `e2e` and `agent-device` from the package's own lockfile. The sample is the first `up` in a worktree, without its `instances` and `clerk` lines. The device is ready when `up` prints the `device` line that ends in `installed `, which is its last line. `run` does the same steps itself, so `up` only starts the slow part early. Teardown is `down` (see [Cleanup](#cleanup)). + +`up` does four things: + +- It builds the dev client when no build matches the native inputs: `turbo build` for the three packages, `expo prebuild --clean`, then `xcodebuild` or `gradlew assembleDebug`. The build overwrites the test app's generated `package.json`, `ios/`, and `android/`. A change to anything else reuses the build and prints `build local reused`. +- It creates this worktree's Clerk application through Clerk's Platform API, in the team's verification workspace, and puts its development instance on the standard settings in `src/core/instances/base.json`. The application holds only the users that this worktree's runs create, and `down` deletes it. [Test instances](references/instances.md) has the credential lookup, the application's lifetime, and its limits. +- It leases a lane, which is a simulator or emulator that the CLI creates, and installs the build. +- It starts `tsdown --watch` in `packages/expo` (the watch build) and `expo start` on the lane's Metro port. + +`up` is idempotent. It keeps a lease that this worktree already holds. A failed `up` stops the Metro and the watch build that it started. + +On a local device a JS change reaches the app with no build. Before the specs start, `run` waits until the watch build has caught up and Metro serves the current code, and it fails with `NOT_READY` and the path of the Metro log when Metro never does. [How a change reaches the app](references/freshness.md) has the native inputs, the checks, the ports, and the logs. It also says what to do after a change to another workspace package, such as `@clerk/clerk-js` or `@clerk/shared`. While Metro runs, never run `pnpm --filter @clerk/expo build` or a build of a package that `@clerk/expo` depends on, because the build deletes the `dist` that Metro serves. + +A worktree can hold one lane of each platform. The two lanes share the watch build and the application, and each has its own Metro. Run them one after the other. The application serves one run at a time, because a run can change its settings. A `run` that starts while a run on the other platform is driving leases its device and then fails with `DEVICE_BUSY`, and its fix is to let that run finish and rerun. + +A Mac has four iOS lanes and two Android lanes, shared by every worktree on it. When all are taken, `up` and `run` fail with `POOL_FULL`, and `--wait ` on either verb waits for a lane. The CLI drives only the simulators and emulators that it creates. [Local devices](references/devices.md) says how to find a lane's UDID or serial. + +The test app is built locally on macOS only. On a Linux machine that can run the emulator, the CLI picks the local backend for Android, and the build then fails with `UNSUPPORTED`. + +## Doctor + +```console +$ integration/expo-native/bin/control-clerk-expo doctor --platform ios +$ integration/expo-native/bin/control-clerk-expo doctor --platform android +``` + +Run it first, and again whenever anything looks off. Without `--live` it only reads. It creates no file, no device, and no Clerk application. Each line starts with `ok`, `warn`, `skip`, or `FAIL`, then has the id of the check and what the check found. `skip` marks a check that did not run, and its text starts with `not run:`. A failing check also prints a `fix:` line with the command to run, and `doctor` exits 3. A warning does not change the exit code. + +After the once-per-machine setup and before the first `up`, `build` is the one failing check, and its fix is the `up` command for that platform. A machine with no Platform API credential fails `instances`, and the fix line says how to supply one. `doctor --live` also creates one application, configures it, compares it with the standard settings, and deletes it, which proves that the credential can do each. + +## Drive + +Input reaches the app only through specs. A spec is a TypeScript file that uses the `host` fixture from `specs/fixtures.ts` and e2e's `screen` locators. + +```console +$ integration/expo-native/bin/control-clerk-expo run native-auth-view --platform ios # one feature: every spec in specs/golden/native-auth-view/ +$ integration/expo-native/bin/control-clerk-expo run custom-flow-sign-in/complete # one golden spec, on iOS +$ integration/expo-native/bin/control-clerk-expo run specs/explored/.e2e.ts # a spec you wrote +$ integration/expo-native/bin/control-clerk-expo run native-auth-view native-js-sync # several targets in one run, with one video +$ integration/expo-native/bin/control-clerk-expo screen --platform ios # the UI tree that is on screen now; --png adds a screenshot +``` + +`run` also takes `--grep `, `--retries `, `--no-video`, and `--wait `. `--retries ` runs a failed test again, up to `n` more times. The default is 0, so a run of your own change shows exactly what happened. The wait covers a free lane and another `run` in this worktree that holds the device. A spec limited to one platform with `test(title, { platforms: ['ios'] }, fn)` reports as skipped on the other. + +For a JS change on a local device, edit the source and `run` the spec, with no `up` in between. For a change to a native input, the same `run` rebuilds the dev client first. + +### Sign in with the form or with a ticket + +A change that touches sign-in or sign-up gets a spec that drives the real form with a `+clerk_test` identity and the test code. Any other change reaches a signed-in state with a sign-in ticket, `host.launch({ signedInAs: user })`. + +Type only test identities: `+clerk_test` emails, phone numbers 555-0100 to 555-0199, and the code `424242`, which specs import as `CLERK_TEST_CODE`. [The feature map index](features/README.md) lists the test identities and the identifier for each step of the forms. + +### Write a spec + +```ts +import { test, expect } from '../../fixtures.ts'; +import { nativeProfile, nativeUserButton } from '../../native.ts'; + +test('the home UserButton opens the profile of the signed-in user', async ({ host, screen, platform }) => { + const user = await host.seedUser(); + await host.launch({ signedInAs: user }); + await host.tap(nativeUserButton(screen, platform)); + await host.tap(nativeProfile(screen).manageAccount); + await expect(screen.getByText(user.email)).toBeVisible({ timeout: 20_000 }); + await host.screenshot('profile'); +}); +``` + +- `host.seedUser({ phone?, password? })` creates a `+clerk_test` user in the worktree's application. With `password: true` the user also has a password that the run generated for that user, and `user.password` holds it. Pass it to `host.fill`, as `host.fill(field, user.password!)`, and to nothing else. It prints as ``, `run` redacts its value from the logs it writes, and `attach` refuses a run whose files hold it. For a sign-up through the form, `host.newEmail()` returns a new address of this run, and `host.newPhone()` returns a 555-01xx number that no user holds. `host.newEmail()` calls nothing and records nothing. After the tests, `run` finds the user by the run's prefix in the address. +- `host.launch({ signedInAs?, authMode?, keepStorage?, landsOn? })` relaunches the test app and waits up to 60 seconds for its home. Every launch opens on the home, and a spec taps from there to the screen it needs, as a user does. `authMode` is the mode of every AuthView the home opens: `signIn`, `signUp`, or `signInOrUp`. A launch starts with fresh storage unless `keepStorage` is true, so a launch with no `signedInAs` is signed out and lands on the home's `Sign in` button. `host.launch()` with no argument is that launch. A launch with `signedInAs` lands on the home, which must show that user's email and user ID. When the home should show anything else, pass its locator as `landsOn`. A launch that keeps storage and has no `signedInAs` needs it. `host.launch` fails at once when the test app shows its error screen, and the error has the message on that screen. +- The home lists a link to each of the test app's other screens. Signed out, it has `Sign in` (`host.app.signIn`, AuthView in a modal), `Sign in with a logo` (the same modal with a React Native logo), `Sign in full screen` (`host.app.signInFullScreen`, AuthView with no close button), `Sign in full screen with a close button`, `Custom sign-in`, `Custom sign-up`, and `Native modules`. Signed in, it has `Embedded profile` (UserProfileView inline). `Token cache` shows in both states. `homeLinks(screen)` in `specs/native.ts` has the locator of each link that `host.app` does not hold. A new screen goes in `screens/destinations.ts` and `verify/VerifyHost.tsx` of the test app and in `homeLinks`. +- `host.expectSignedInAs(user, timeoutMs?)` waits for the home to show `Signed in as `, the user's ID, and a session ID. Pass the email alone for a user that the sign-up form created. `host.expectSignedOut(timeoutMs?)` waits for the home to show `Signed out` with no user ID and no session ID. Use them for the outcome of a flow that returns to the home. When one of these waits or the wait of `host.launch` runs out, its message says what the home showed at the last reading, how many passes and screen reads the wait made, how long the slowest read took, and the error of a read that failed. +- `host.app` has the locators of the home: `signIn`, `signInFullScreen`, `signedOut`, `signedIn`, `userId`, `sessionId`, `signOut`, and `error` for the error screen. `host.runId` is the id of the run, for a password that must be new in each run. +- `host.tap(locator)` is `locator.tap()` with the assertion timeout, and `host.fill(locator, text)` taps a field and types into it. When the first tap left no field focused, it taps the field once more before it types. Fill the text fields of the native views with `host.fill`. On iOS a native text field shows no text input until it has focus, so a plain `locator.fill()` on it fails with "no text input found at the provided coordinates to clear". After it types, `host.fill` reads the focused text input back. When the screen tree marks no text input as focused, as it does on iOS for a while after a slow snapshot, it reads the value that the node the locator names shows. It sends a long text sixteen characters to a command, because the iOS runner abandons a command after 30 seconds. If the input is still empty after two seconds and the screen shows no text that it did not show before the typing, `host.fill` taps it and types once more, and it fails with "the text never reached the field" if the input is empty again. A screen that shows new text has taken the typed text and moved on, as a code screen does when its code is complete, so `host.fill` types nothing more there. For this it compares the letters of the screen's text only, so the seconds that a code screen counts down beside `Resend` are not new text. If the input holds something other than the text, `host.fill` replaces the input's contents with the text once, and it fails with "the field does not hold the typed text" if the input is still wrong. It compares letters and digits only, so the formatting a field adds does not count. A password field withholds its value, so a password is typed once and is not confirmed. When the iOS runner refuses a tap, a typing command, or a screenshot because it is still finishing an earlier command, `host.tap`, `host.fill`, and `host.screenshot` wait five seconds and try the step again, up to six times, and then fail with the refusal. After a refused typing command, `host.fill` replaces the input's contents with the text and does not type that command again, so characters that did land are not typed twice. On iOS an empty React Native field of the test app reads as its placeholder, which `host.fill` treats as any other value that is not the text. +- `host.screenshot(label)` writes `screenshots/