FE-1663: Experiment Live Full-Duplex Migration

[kostandinang]

Note

Experiment 1: a standalone GPT-Live interview using the existing Voice controls.
Local testing passed; Vercel preview review is pending for pacing, interruptions, control placement and End behavior.
No application changes are executed. Domain-specific Brunch integration is in Experiment 2 (#9673).

🌟 What is the purpose of this PR?

Test whether GPT-Live-1 natively improves process interviewing: concise relevant
follow-ups, room for hesitation, and interruption/correction without losing meaning.
Smooth audio alone is not acceptance.

This first experiment is a labelled standalone conversation through the existing
Voice UX, not Brunch output or application execution. It asks whether the person
feels heard and the interview advances; full Brunch integration is deferred.

🔗 Related links

• FE-1663 (internal)
• FE-1661 (internal; retained migration assessment)
• Child experiment FE-1664 / #9673, stacked on this branch
• Official Live migration guide
• Official Live prompting guide

🚫 Blocked by

• Manual conversation/media acceptance. Keep this PR in draft.

🔍 What does this change?

• Adds an isolated Live controller and website session endpoint using existing
  Voice enablement/credentials. Creation starts WebRTC; wait for session.started
  without sending another session.start.
• Gives Live a purpose-led process-interviewer prompt structured around the official
  backchannel, interruption and delegation policies. It explicitly has no backend
  capabilities and cannot claim application execution or saved changes.
• Reuses Voice consent and the dock, with experimental wording only at consent.
  Local WebRTC audio levels drive Listening/Speaking and the microphone ribbon.
  Unsupported lifecycle/playback actions stay hidden; no separate experiment panel.
• Keeps transcript deltas and delegation metadata out of canonical submission.
  Local media stops immediately on End; remote closure is not guaranteed.
• Selects realtime or live through PETRINAUT_VOICE_PROVIDER, pinned for the
  mounted conversation. Unset remains Realtime; its prompts, admission, queue,
  settlement and playback remain unchanged. Live cannot submit transcripts,
  execute tools or insert canonical history. No automatic retry or input replay.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• modifies an npm-publishable library and Kostandin has added a changeset file

Adds a patch changeset for @hashintel/petrinaut.

📜 Does this require a change to the docs?

The changes in this PR:

• require changes to docs which are made as part of this PR

Updates the website README, mission and assistant user guide. Existing Realtime
screenshots remain applicable; experimental dock captures are local review evidence.

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• do not affect the execution graph

⚠️ Known issues

• Kostandin manually tried the standalone experiment and reported that it looked
  fine; the exact tested revision was not supplied. This supports proceeding, not
  comprehensive acoustic, naturalness or Brunch compatibility acceptance.
  Model instructions are guidance, not an output guarantee.
• Lost connections can leave remote closure unconfirmed. No transcripts or recordings
  are retained. Dock activity is not a turn boundary or proof of heard playback.
  Browsers without audio-level telemetry retain Listening without an input meter.
• Existing unauthenticated Voice endpoint risk applies (FE-1622); keep the experiment local.
• The forced Turbo gate hit a sandbox restriction on tsx's Unix socket. Equivalent
  package checks ran directly against prepared dependencies with external network denied.
• A pre-commit website run failed six tests while generated Petrinaut build chunks
  changed underneath it. Rerunning against the settled build passed all 518 tests.

🐾 Next steps

Draft stack: main → FE-1663 / #9671
→ FE-1664 / #9673.
Domain-specific elicitation details, modelling rules and application execution are
deferred to experiment 2, Experiment Live Full Brunch Integration (see the
preserved integration scope).
This experiment tests general interviewing and native conversational behavior only;
it does not port Brunch's domain prompt or grant Live domain authority in experiment 2.
The second experiment's authority recut and implementation live in the child PR,
not this standalone comparison.

PR 2 must retain finalized input → composer → Flue → Brunch → authorized Petrinaut
execution → complete-turn settlement → frozen canonical reply → Live delivery.
Brunch alone authors domain answers/questions. Approval, settlement and playback
eligibility remain distinct; unknown outcomes are not replayed or spoken as success.

The child records the accepted experimental policy in a separate authority commit:
authoritative transcription plus native, unbuffered Live delivery with best-effort
speech control. Settlement gates supplied Brunch context, not all audible speech;
this is neither strict output gating nor exact relay. The child's reported connection
failure remains unresolved, and no live integrated exchange is claimed. Standalone
success does not reopen FE-1624. This PR adds neither an audio evaluation harness
nor PR 2 integration.

🛡 What tests cover this?

• Provider-free checks passed: 883 Petrinaut tests, 520 website tests, both package
  builds and typechecks. Lint has zero errors and one pre-existing Realtime effect
  warning. Formatting and diff checks passed.
• New mocked tests cover default/pinned provider selection, client-delegated
  handshake, consent, canonical isolation, optional dock actions, failure and
  cleanup. Activity tests cover simultaneous input/output, silence, unavailable
  telemetry and samples arriving after Stop. Realtime retains its regression coverage.
• Browser checks covered consent, mocked connection/error, collapse/expand, End,
  fresh consent, Exit and unchanged Realtime setup. Screenshots were inspected.
  The real Live controller was also exercised with mocked media and SDP to verify
  Listening/Speaking transitions and End cleanup in the existing dock.
  Verification made no real microphone or provider requests.

Package commands: test:unit (run for Petrinaut), lint:tsc, lint:eslint, build.
Kostandin's manual conversational/media acceptance remains outstanding.

❓ How to test this?

From this branch's repository root, with OPENAI_VOICE_API_KEY exported or in the
website's .env.local (full preparation commands are in the website README):

PETRINAUT_OPENAI_VOICE_ENABLED=true PETRINAUT_VOICE_PROVIDER=live yarn dev:brunch:panel

Open http://localhost:4915/new → AI panel → Start voice mode → consent → Start voice.
Only the final action activates microphone/provider. Allow 10–15 minutes:

1. Explain a familiar process and allow follow-ups.
2. Hesitate, pause, answer briefly, then elaborate.
3. Interrupt mid-sentence and correct an earlier detail.
4. Speak during output; note lost meaning, verbosity and acknowledgements.
5. End voice mode; verify microphone indicator and speaker silence. Stop the dev
   command, then start Realtime with the existing Brunch backend environment:

PETRINAUT_OPENAI_VOICE_ENABLED=true PETRINAUT_VOICE_PROVIDER=realtime yarn dev:brunch

Reload for a new session. Judge feeling heard/interview progress separately from
transport correctness. Prior experiments do not constitute a passing baseline.

📹 Demo

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
petrinaut	Ready	Preview	Sep 11, 2026 5:32pm UTC
petrinaut-docs	Ready	Preview	Sep 11, 2026 5:32pm UTC

2 Skipped Deployments

Project	Deployment	Actions	Updated
hash	Ignored	Preview	Sep 11, 2026 5:32pm UTC
hashdotdesign-tokens	Ignored	Preview	Sep 11, 2026 5:32pm UTC

[cursor]

PR Summary

Medium Risk
Introduces a billable Live voice API path and WebRTC session lifecycle alongside existing Realtime; Realtime/Brunch paths are intended to stay isolated but voice enablement and same-origin endpoint risks still apply.

Overview
Adds an optional GPT-Live-1 experiment (PETRINAUT_VOICE_PROVIDER=live) alongside unchanged Realtime: a new /api/voice/live-session handler creates client-delegated WebRTC sessions with a standalone process-interview prompt and strict request validation; the browser createLiveConversation path waits for session.started, ignores transcript/delegation events, drives Listening/Speaking from local WebRTC stats, and handles disconnect recovery and bounded cleanup without retries or canonical chat submission.

Voice UX reuses consent and the dock via LiveConversationControl (experimental disclosure, provider pinned per conversation mount). @hashintel/petrinaut treats host voice actions as optional so the dock hides mute, reconnect, resume, and playback menus when unsupported; adds a neutral connected phase, measures collapsed dock height so viewport controls sit above the compact dock instead of reserving full panel width, and documents the experimental flow.

^{Reviewed by Cursor Bugbot for commit ef2643a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit b72c2ac. Configure here.}