fix: keep backup codec track clones in sync with the published track

[ydiarra]

The bug

With a backup codec (e.g. VP9 primary, VP8 backup through backupCodec / backupCodecPolicy: SIMULCAST), addSimulcastTrack() gives each backup codec its own this.mediaStreamTrack.clone(), kept in SimulcastTrackInfo.mediaStreamTrack. Only restartTrack() renewed that clone. replaceTrack() goes through LocalTrack.setMediaStreamTrack(), which swaps the primary sender and then calls resumeUpstream(), and LocalVideoTrack.resumeUpstream() puts sc.mediaStreamTrack, the clone taken at publish time, back on every backup sender.

As a result:

• After replaceTrack(newTrack), with or without a pauseUpstream() before it, backup codec subscribers keep receiving the old track. Once the old track stops producing frames, they get no video at all, while primary codec subscribers are fine.
• The clone is never stopped, so the capture stays open after the app has stopped the original track: the camera stays on.

Found by reading the code, not reproduced:

• mute() on an SDK managed camera stops _mediaStreamTrack to release the camera, but leaves the backup clones running, so the camera is never released while muted. On unmute, restart() goes through setMediaStreamTrack() too and the backup goes back to the old clone, so a setDeviceId() done while muted never reaches backup codec subscribers.
• setProcessor() puts processedTrack on the backup senders, but sc.mediaStreamTrack is still the unprocessed clone, so the next resumeUpstream() (e.g. after a mute event on the track) sends the unprocessed track on the backup codec.
• unpublishTrack() clears simulcastCodecs without stopping the clones. With stopOnUnpublish = false, which republishAllTracks() uses after a full reconnect, the clones leak.

We hit this in production in WorkAdventure, which publishes VP9 with a VP8 backup and replaces its own camera track when the camera is turned back on. A subscriber that could not decode VP9 lost a publisher's camera after the publisher toggled it, while the other subscribers were fine. We currently work around it in workadventure/workadventure#6789 by doing what restartTrack() does after each replaceTrack().

The fix

• LocalVideoTrack.onMediaStreamTrackChanged(), the hook setMediaStreamTrack() already runs on replaceTrack(), restart() (so unmute and restartTrack()) and stopProcessor(), now gives each backup codec a fresh clone of the current track, sends it unless upstream is paused, and stops the clone it replaces. restartTrack() no longer needs its own loop.
• setProcessor() uses the same helper, so sc.mediaStreamTrack stays the track the backup sender actually sends.
• mute() on an SDK managed camera also stops the backup clones, so the camera is released.
• unpublishTrack() stops the clones before dropping them, which also covers republishAllTracks().

Testing

• Two new tests in LocalVideoTrack.test.ts, both failing on main: pause + replaceTrack() sends a clone of the new track on the backup sender and stops the old clone; muting an SDK managed camera stops the backup clone.
• pnpm test (977 passed), tsc --noEmit, eslint and prettier --check are clean.
• The repro below, in Chrome for Testing 153.0.8010.12 (headless, macOS 27.0.1), on main at b6b9210 (2.23.0) and with this branch. The setProcessor() path has no dedicated test; unpublishTrack() has one in LocalParticipant.test.ts, including after a disconnect.

	backup codec frames/s after the camera toggle	first clone
main	0	live
this branch	30	ended

Same numbers with a pauseUpstream() before the replaceTrack().

Repro (no server needed)

It publishes a LocalVideoTrack over a loopback RTCPeerConnection, with the backup sender wired up the way publishAdditionalCodecForTrack() does it. Each "camera" is a canvas that stops drawing when it is turned off.

import { LocalVideoTrack } from 'livekit-client';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
function camera() {
  const canvas = Object.assign(document.createElement('canvas'), { width: 320, height: 180 });
  const ctx = canvas.getContext('2d');
  let n = 0;
  const timer = setInterval(() => {
    ctx.fillStyle = `hsl(${(n += 7) % 360},60%,50%)`;
    ctx.fillRect(0, 0, 320, 180);
  }, 33);
  const track = canvas.captureStream(30).getVideoTracks()[0];
  return { track, off: () => { clearInterval(timer); track.stop(); } };
}
async function framesSent(sender) {
  let frames = 0;
  (await sender.getStats()).forEach((s) => { if (s.type === 'outbound-rtp') frames += s.framesSent ?? 0; });
  return frames;
}

const cam1 = camera();
const track = new LocalVideoTrack(cam1.track, undefined, true);
await sleep(50);

const pc1 = new RTCPeerConnection();
const pc2 = new RTCPeerConnection();
pc1.onicecandidate = (e) => e.candidate && pc2.addIceCandidate(e.candidate);
pc2.onicecandidate = (e) => e.candidate && pc1.addIceCandidate(e.candidate);
track.sender = pc1.addTransceiver(cam1.track, { direction: 'sendonly' }).sender;
const info = track.addSimulcastTrack('vp8');
const backup = pc1.addTransceiver(info.mediaStreamTrack, { direction: 'sendonly' });
info.sender = backup.sender;
await pc1.setLocalDescription(await pc1.createOffer());
await pc2.setRemoteDescription(pc1.localDescription);
await pc2.setLocalDescription(await pc2.createAnswer());
await pc1.setRemoteDescription(pc2.localDescription);
await sleep(1500);
const firstClone = info.mediaStreamTrack;

// camera off, then on again with a new track
cam1.off();
await sleep(500);
await track.replaceTrack(camera().track);
await sleep(500);
const before = await framesSent(backup.sender);
await sleep(1000);
console.log({
  backupFramesPerSecond: (await framesSent(backup.sender)) - before,
  firstCloneReadyState: firstClone.readyState,
});

[changeset-bot]

🦋 Changeset detected

Latest commit: 69266e1

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
livekit-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[CLAassistant]

All committers have signed the CLA.

[lukasIO]

thanks for the PR!
The issue is real and the fix is mostly there already.

However the failure path should not keep the previous clone alive. If the backup sender rejects camera B, it should stop sending camera A:

await sc.sender.replaceTrack(null);
previousTrack.stop();
newTrack.stop();

The backup entry should then be marked as needing refresh. A later replaceTrack(B) must bypass the same-track early return and retry the backup sender with a fresh clone of B.

Otherwise, camera A can continue being published after the application tried to replace it, and retrying with B can resolve without updating the backup sender.

Could we add a regression test covering this sequence?

1. Replacing A with B fails on the backup sender.
2. The backup sender receives replaceTrack(null).
3. A’s clone is stopped.
4. Retrying with the same B track sends a fresh B clone successfully.

[devin-ai-integration]

Devin Review found 1 new potential issue.

[devin-ai-integration]

🔴 Late retry blanks the backup codec

When another camera replacement starts during retryAfterTrackChange, the retry can finish after the newer backup swap. refreshSimulcastTrack stops the older clone, so the late retry leaves the backup sender publishing an ended track.

Learn more

A rejected backup sender sets needsRefresh and waits for a new clone. The queued retry acquires the track-change lock but releases it before awaiting refreshSimulcastTrack. A later replaceTrack can refresh that same sender while the retry's replaceTrack is pending. If the older sender operation finishes last, it attaches a clone the later refresh has stopped. The retry cleared needsRefresh before awaiting, so nothing requests another swap.

Example: Camera B's backup swap fails. The retry starts swapping a fresh B clone, then camera C replaces B and successfully swaps a C clone. If the B swap completes last, the backup sender ends up on B's stopped clone instead of C.

Recommended fix: Keep backup refreshes serialized with track changes until the sender swap settles. Also serialize or recheck against upstream pause and stop operations before committing an asynchronous retry.

---

Was this helpful? React with 👍 or 👎 to provide feedback.