Skip to content

pi: live view, viewer chat, and human handoff via browser_handoff - #74

Open
ekenberg wants to merge 1 commit into
BetterWright:mainfrom
ekenberg:feat/pi-live-view
Open

pi: live view, viewer chat, and human handoff via browser_handoff#74
ekenberg wants to merge 1 commit into
BetterWright:mainfrom
ekenberg:feat/pi-live-view

Conversation

@ekenberg

Copy link
Copy Markdown
Contributor

The live-view docs list a surface for every agent host — the standalone
harness gets live_view/handoff tools, MCP clients get browser_handoff
but the native Pi extension had none. On a headless box (my case: a VPS
driven by Pi over Tailscale) that leaves no way to compose "the agent drives
its persistent in-process session" with "a human watches, coaches, or takes
over", because betterwright view attaches to the daemon sessions, not the
extension's private worker.

This registers a browser_handoff tool in the Pi extension, modeled on the
MCP server's handler, with one Pi-specific addition each way:

  • Actions start / status / stop, plus a blocking wait backed
    by waitForHandoff() — the viewer elevates with the reason and Done /
    Cancel, and the tool resolves when the human finishes. A timeout resolves
    as a normal {action: "timeout"} result (resumable), never a tool error;
    an abort is honored without leaking listeners.
  • Viewer chat reaches the model between turns. While a view runs, the
    extension polls liveViewDrainChat() (default 3 s,
    BETTERWRIGHT_PI_CHAT_POLL_SECONDS, exponential backoff on transient
    failures, unref'd timers) and delivers messages via Pi's
    sendMessage({deliverAs: "followUp", triggerTurn: true}) — so a message
    typed in the dock wakes an idle agent instead of waiting for the next tool
    call. Browser-step notes mirror into the dock via liveViewPostChat, as
    the MCP server does per call.
  • Same deployer opt-in as MCP: a bind that reaches beyond the machine
    (lan/tailscale) requires BETTERWRIGHT_LIVE_VIEW=1; expose: "local"
    never does. liveViewFromEnv and isLoopbackHost move to
    live-view-config.ts so both surfaces share them (re-exported from
    mcp-server, their public home — no caller changes).
  • status strips the capability token and URL, exactly like MCP;
    re-issuing start re-shares the same URL (useful after a lost terminal
    scrollback — the view survives worker restarts on the same port+token).
  • browser_handoff is deliberately not in BROWSER_TOOL_NAMES: a
    pending handoff (or a user still watching) must survive step-budget
    exhaustion.

Divergence from MCP worth a reviewer's eye: the reason parameter is
surfaced during wait (the viewer banner) rather than appended to the
start response text, since Pi has a real blocking handoff instead of
status-polling.

Testing

  • npm run release:check green (lint, typecheck ×3, build, unit tests,
    hand-written types, package).
  • New unit tests: start/status/stop/wait against a live-view fake (URL
    relay, token/URL stripping, idempotent re-start, chat delivered as
    follow-up guidance and stopped with the view, opt-in gating incl.
    expose: local/tailscale, wait outcomes done/timeout, budget-exhaustion
    survival).
  • Verified end-to-end on a headless VPS with Pi over Tailscale: watch,
    chat-wakes-idle-agent, take-control handoff round-trip, tmux
    detach/reattach, viewer close/reopen on the same URL, and a worker
    SIGKILL revived the view on the same port+token with the viewer
    auto-reconnecting.

Register a browser_handoff tool in the native Pi extension so a Pi agent
can share the live viewer (start/status/stop) and pause for human hands
(wait, backed by waitForHandoff) on its own in-process session. Start
mirrors the MCP server's gating: a non-loopback bind requires the
deployer opt-in BETTERWRIGHT_LIVE_VIEW=1; expose "local" never does.
liveViewFromEnv and isLoopbackHost move to live-view-config.ts so both
surfaces share them (re-exported from mcp-server, their public home).

While a view runs, freeform viewer-dock messages are polled
(BETTERWRIGHT_PI_CHAT_POLL_SECONDS, default 3 s, error backoff, unref'd
timers) and delivered as follow-up user guidance that wakes an idle
agent; browser-step notes mirror back into the dock. Status strips the
capability token and URL; re-issuing start re-shares the same URL, and
a handoff timeout resolves as a normal, resumable result.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant