Replace replay live view with headed browser + agent overlay#86
Merged
aidenybai merged 172 commits intofirst-minorfrom Apr 7, 2026
Merged
Replace replay live view with headed browser + agent overlay#86aidenybai merged 172 commits intofirst-minorfrom
aidenybai merged 172 commits intofirst-minorfrom
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
aidenybai
force-pushed
the
headed-overlay-video
branch
from
1b092a0 to
c3fceaf
Compare
commit: |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
3 issues found across 54 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/browser/src/runtime/overlay/index.tsx">
<violation number="1" location="packages/browser/src/runtime/overlay/index.tsx:562">
P1: This recursive bootstrap can overflow the stack before the overlay finishes mounting, so the first cursor update may fail on a fresh page.</violation>
</file>
<file name="apps/cli/package.json">
<violation number="1" location="apps/cli/package.json:43">
P1: `|| true` masks failures from the entire build pipeline, so `vp pack` can fail while the script still exits successfully.</violation>
</file>
<file name="packages/browser/src/mcp/server.ts">
<violation number="1" location="packages/browser/src/mcp/server.ts:48">
P3: Restrict comment extraction to actual comment lines; the current pattern also matches `//` inside URLs (for example `https://...`), which generates incorrect cursor labels.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
| }); | ||
| }); | ||
|
|
||
| const pushStepEvent = Effect.fn("McpSession.pushStepEvent")(function* (state: ViewerRunState) { |
Contributor
There was a problem hiding this comment.
videoSegmentsRef not reset when opening a new session, causing video segments from previous sessions to be concatenated with current session's video
✅ Resolved — addressed in the latest commits on this branch.
Contributor
There was a problem hiding this comment.
2 issues found across 4 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/browser/src/runtime/overlay/index.tsx">
<violation number="1" location="packages/browser/src/runtime/overlay/index.tsx:429">
P2: Guarding only the `click` event no longer fully blocks page interaction; `mousedown`/`pointerdown` side effects can still fire before this handler runs.</violation>
<violation number="2" location="packages/browser/src/runtime/overlay/index.tsx:494">
P2: The `computeRects` rAF loop calls `setHighlightRects(rects)` with a new array reference on every frame (~60fps), triggering a full React re-render of `AgentOverlay` each time even when highlight positions haven't changed. Compare previous and next rects before calling the setter to avoid unnecessary re-renders.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.
…romises NodeServices.layer is already provided via Layer.provideMerge so FileSystem is available in the runtime context. Use it for the performance trace mkdir + writeFileString instead of raw fs/promises.
Replace sync existsSync-based wallpaper resolution with an Effect.fn that yields FileSystem.exists. Path candidates are still computed at module load (import.meta.url is inherently sync) but existence checks go through Effect's async FileSystem service. Resolved once during McpSession.make and stored for the session lifetime instead of being a module-level constant.
…bort native Promises
Playwright's browser.close() and pageVideo.path() can hang indefinitely on slow Windows CI runners. Promise.race with a setTimeout lets close() stop waiting and proceed — the underlying Promise still runs to completion in the background since native Promises can't be interrupted without AbortSignal support.
- Switch to namespace imports (import * as fs) for node:fs across add-skill, expect-skill, video-processor, and video-processor tests - Inline resolveBrowserHeaded — trivial one-liner doesn't need its own function
@ffmpeg-installer/ffmpeg downloads ~70MB of platform-specific ffmpeg
binaries at install time, making npx installs painfully slow. Replace
with which.sync("ffmpeg") to find the system-installed ffmpeg. If
ffmpeg isn't available, video processing gracefully degrades to file
copies (no framing/concat).
- Moved from dependencies to optionalDependencies so it doesn't block npx installs when the binary download fails or is slow - resolveFfmpegPath now tries system ffmpeg (via which) first, only falling back to @ffmpeg-installer/ffmpeg if no system binary found - If neither is available, video processing gracefully degrades to plain file copies
Removes the ~70MB bundled ffmpeg binary that was downloaded at install
time. resolveFfmpegPath now uses which.sync("ffmpeg") to find the
system-installed binary. If ffmpeg isn't on PATH, video processing
gracefully degrades to plain file copies.
- Add @ffmpeg/ffmpeg, @ffmpeg/util, @ffmpeg/core as dependencies - resolveFfmpegPath probes system ffmpeg first (fast path), falls back to wasm ffmpeg via lazy dynamic import when not found - runWasmFfmpeg reads input files into wasm virtual filesystem, runs the command, and writes output back to disk - No native binary downloads at install time — @ffmpeg/core is pure wasm loaded lazily at first use - Remove @ffmpeg-installer/ffmpeg entirely
- runWasmFfmpeg: wrap ffmpeg lifecycle in try/finally so terminate() runs on both success and error paths (prevents wasm instance leaks) - runWasmFfmpeg: use Map for file name resolution instead of nested loop with exact string matching - Cache wasm failures — set mode to "none" after first wasm error so subsequent calls fail fast instead of re-loading the 30MB wasm - Move BROWSER_CLOSE_TIMEOUT_MS and VIDEO_PATH_TIMEOUT_MS to constants.ts; reuse withTimeout helper for video path resolution
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 37723c4. Configure here.
| "@ffmpeg/core": "0.12.10", | ||
| "@ffmpeg/ffmpeg": "0.12.15", | ||
| "@ffmpeg/util": "0.12.2" | ||
| } |
There was a problem hiding this comment.
Root monorepo package.json gains heavyweight production dependencies
Medium Severity
The monorepo root package.json now has a dependencies block with @ffmpeg/core, @ffmpeg/ffmpeg, and @ffmpeg/util. These are already declared as dependencies in packages/browser/package.json. Adding them as production dependencies (not devDependencies) at the workspace root is atypical—it forces installation of ~30MB+ of WASM binaries for every workspace consumer, including packages that never use ffmpeg. This likely belongs only in packages/browser/package.json (which already has them) or, if needed for the CLI's neverBundle externalization, in apps/cli/package.json.
Reviewed by Cursor Bugbot for commit 37723c4. Configure here.
Member
Author
There was a problem hiding this comment.
✅ Resolved — addressed in later commits or file deleted.
Video framing was causing 2+ minute close times and broken video output. The raw Playwright video is now copied directly to /tmp. Removed: video-processor.ts, wallpaper.webp, mixed-content.webm fixture, @ffmpeg/core, @ffmpeg/ffmpeg, @ffmpeg/util, FFMPEG_TIMEOUT_MS, FRAME_PADDING_PX, CLI wallpaper copy build step, neverBundle entries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--headlessflag to opt out)Test plan
pnpm typecheckpasses across all 8 packagespnpm buildsucceedsexpect -m "..." -y --headedopens browser with overlay, generates .webm with wallpaperNote
Medium Risk
Removes the existing rrweb/live-replay and related artifacts in favor of a new headed/overlay + video-only flow, which changes core execution UX and artifact outputs across CLI, CI, and browser packages. Risk is mainly behavioral/regression risk in run options defaults and artifact generation/consumption, not security-sensitive logic.
Overview
Shifts the CLI/browser execution experience away from rrweb-based live replay and HTML/NDJSON artifacts to a headed Playwright-first workflow with a new shadow-DOM agent overlay and video as the primary artifact.
CLI options and UI are updated accordingly:
--headed/--replay-hostare removed in favor of--headless, headed mode becomes the default for interactive runs, and results/reporting surfaces drop replay URLs (Results screen and GitHub Actions outputs now only reference video + screenshots).On the browser side, the rrweb/live-view/proxy/replay machinery and related errors/exports are deleted, video recording is resized to 1080p, overlay injection/cleanup is added to MCP session lifecycle (with timeouts and more robust network request status tracking), and a new
OverlayControllerservice is introduced to drive cursor/label/highlight behavior. Build tooling is expanded to bundle Tailwind/React overlay CSS (including a rem→px post-process) and new ffmpeg-related deps are added.Reviewed by Cursor Bugbot for commit 577def2. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by cubic
Replaces
rrweblive replay with a headed Playwright run and a shadow‑DOM agent overlay. Outputs a single 1080p.webmvideo (optional wallpaper) plus screenshots; local CLI defaults to headed (--headlessto opt out), CI/agents auto‑detect and run headless.New Features
ref()or parsed CSS selectors (querySelector,locator,getByRole,getByText) and persists through scroll/resize via@medv/finder..webm; video processor can frame with wallpaper viaffmpeg; artifacts simplified to video + screenshots; Results screen shows video only; all replay/live‑view code, proxy, and tests removed.--headlessreplaces--headed; local default is headed, CI/agents auto‑detect and run headless; overlay CSS built with Tailwind (+ rem→px) and inlined via build plugin; Playwright MCP tool accepts a description for overlay labels; add@ffmpeg/ffmpeg,@ffmpeg/util,@ffmpeg/coredeps and exclude them from CLI bundling; CLI build copies wallpaper; newEXPECT_HEADEDenv support.Bug Fixes
ffmpegpath; add timeouts; escape single quotes in concat lists; propagateVideoProcessError; clean temp files on failure.Written for commit 577def2. Summary will update on new commits.