Draft
Conversation
Contributor
Semver Impact of This PR⚪ None (no version bump detected) 📋 Changelog PreviewThis is how your changes will appear in the changelog. This PR will not appear in the changelog. 🤖 This preview updates automatically when you update the PR. |
Contributor
Android (legacy) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| f5fb57c+dirty | 405.82 ms | 423.92 ms | 18.10 ms |
| 9530cff+dirty | 399.24 ms | 427.22 ms | 27.98 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| f5fb57c+dirty | 43.75 MiB | 48.08 MiB | 4.33 MiB |
| 9530cff+dirty | 43.75 MiB | 48.08 MiB | 4.33 MiB |
Contributor
iOS (legacy) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| f5fb57c+dirty | 1195.00 ms | 1190.48 ms | -4.52 ms |
| 9530cff+dirty | 1230.51 ms | 1231.96 ms | 1.45 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| f5fb57c+dirty | 3.38 MiB | 4.73 MiB | 1.35 MiB |
| 9530cff+dirty | 3.38 MiB | 4.73 MiB | 1.35 MiB |
Contributor
iOS (new) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| f5fb57c+dirty | 1246.91 ms | 1241.61 ms | -5.30 ms |
| 9530cff+dirty | 1220.98 ms | 1216.18 ms | -4.80 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| f5fb57c+dirty | 3.38 MiB | 4.73 MiB | 1.35 MiB |
| 9530cff+dirty | 3.38 MiB | 4.73 MiB | 1.35 MiB |
Contributor
Android (new) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| f5fb57c+dirty | 444.45 ms | 473.36 ms | 28.91 ms |
| 9530cff+dirty | 395.80 ms | 426.40 ms | 30.60 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| f5fb57c+dirty | 43.94 MiB | 48.94 MiB | 5.00 MiB |
| 9530cff+dirty | 43.94 MiB | 48.94 MiB | 5.00 MiB |
antonis
force-pushed
the
fix/e2e-stable-checks
branch
from
05eb0bd to
605f13a
Compare
antonis
changed the title
antonis
force-pushed
the
fix/e2e-stable-checks
branch
2 times, most recently
from
0d6c6da to
60075ed
Compare
Replace retry-based approach (PR #5830) with deterministic fixes: ### Simulator stability (Cirrus Labs Tart VMs) - `wait_for_boot: true` / `erase_before_boot: false` on simulator-action - `xcrun simctl bootstatus booted -b` to block until boot completes - Settings.app warm-up for SpringBoard/system service initialization - `MAESTRO_DRIVER_STARTUP_TIMEOUT` bumped to 180s ### e2e-v2 test runner (cli.mjs) - Per-flow process isolation via individual `maestro test` calls - Maestro driver warm-up flow before real tests (non-fatal) - crash.yml runs first so the next flow verifies post-crash recovery - `execSync` → `execFileSync` to avoid shell interpolation - SENTRY_AUTH_TOKEN redaction in debug logs ### Sample application test fixes - Search all envelopes for app start transaction (slow VM delivery) - Sort envelopes by timestamp for deterministic ordering - Allow-list for TTID/TTFD ops (`navigation`, `ui.load`) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
antonis
force-pushed
the
fix/e2e-stable-checks
branch
from
8f483e2 to
2ff292c
Compare
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📢 Type of change
📜 Description
Alternative to #5830 — fixes E2E test flakiness without per-flow retries.
cli.mjs— each Maestro flow runs in its own process, preventing crash cascadelaunchAppafter simulator boot is unreliable on Tart VMsxcrun simctl bootstatus, Settings.app warm-up,MAESTRO_DRIVER_STARTUP_TIMEOUTbumped to 180slaunchTestAppClearverifies post-crash recoverynavigation,ui.load)execSync→execFileSyncto avoid shell interpolationNo test coverage lost. No retries.
#skip-changelog💡 Motivation and Context
iOS E2E tests fail on Cirrus Labs Tart VM runners due to Maestro driver connection issues on first launch, crash cascade in shared sessions, and envelope delivery order on slow VMs.
💚 How did you test it?
CI
📝 Checklist
sendDefaultPIIis enabled🔮 Next steps
Close #5830 if this approach proves stable.