Don't declare the pipeline frozen when EOS arrived before it was requested - #1313
Merged
Conversation
…ested Logs are showing recorded a VP8 tracks which sometimes fail with "pipeline frozen", but the pipeline wasn't frozen, it had already finished. When the publisher left the room, their encoder emitted a degenerate final keyframe (8x8), the muxer refused the caps renegotiation, and the appsrc died with not-negotiated, pushing EOS downstream as its last act. That EOS reached the file sink and posted on the bus before the controller's own end path ran. sendEOS() then installed its sink EOS probes after the EOS event had already passed (so they could never fire) and armed the 30-second watchdog after the bus handler had already tried to stop a timer that didn't exist yet (so nothing could ever stop it). The timer fired, declared the pipeline frozen, and a correctly finalized recording was reported as failed. Logs show this class of shutdown is common and always benign: end-of-track placeholder frames (8x8 VP8, 2x2 H264) trigger the same not-negotiated error-EOS a few times a day across unrelated customers, and every sampled instance coincides with the publisher leaving the room, the frozen verdict is just the unlucky ordering of a race the shutdown usually wins. The fix makes sendEOS() respect the eosReceived fuse: if the bus EOS was already processed, the pipeline is already stopping and there is nothing to arm. The watchdog callback also checks the fuse, closing the narrow window where the EOS lands between the guard and the timer assignment; in that case it falls through to the idempotent Stop() instead of failing the egress. A genuinely frozen pipeline — EOS sent but never reaching the sinks, still fails after eosTimeout exactly as before. The new controller test reproduces the scenario end-to-end with a real pipeline (silence generator → opus → ogg file sink): it sends EOS directly through the pipeline, invokes SendEOS the moment the bus EOS is processed, and asserts the egress completes with a finalized file. Without the guard it fails deterministically with the production signature, EGRESS_FAILED (error: "pipeline frozen").
paulwe
approved these changes
Jul 27, 2026
Contributor
|
AV-sync stats summary: view in run #30343598075 |
boks1971
approved these changes
Jul 28, 2026
|
|
||
| // As soon as the bus EOS is processed (before Close marks the egress | ||
| // complete), invoke the normal end path — mirroring the source-side | ||
| // shutdown that raced the unsolicited EOS in production. |
Contributor
There was a problem hiding this comment.
have been telling Claude to not do references like this. in production does not mean anything for somebody reading this a couple of months from now.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Logs are showing some VP8/H264 track recordings failing with "pipeline frozen", but the pipeline wasn't frozen, it had already finished. When the publisher left the room, their encoder emitted a degenerate final keyframe (8x8), the muxer refused the caps renegotiation, and the appsrc died with not-negotiated, pushing EOS downstream as its last act. That EOS reached the file sink and posted on the bus before the controller's own end path ran. sendEOS() then installed its sink EOS probes after the EOS event had already passed (so they could never fire) and armed the 30-second watchdog after the bus handler had already tried to stop a timer that didn't exist yet (so nothing could ever stop it). The timer fired, declared the pipeline frozen, and a correctly finalized recording was reported as failed.
Logs show this class of shutdown is common and always benign: end-of-track placeholder frames (8x8 VP8, 2x2 H264) trigger the same not-negotiated error-EOS a few times a day, and every sampled instance coincides with the publisher leaving the room, the frozen verdict is just the unlucky ordering of a race the shutdown usually wins.
The fix makes sendEOS() respect the eosReceived fuse: if the bus EOS was already processed, the pipeline is already stopping and there is nothing to arm. The watchdog callback also checks the fuse, closing the narrow window where the EOS lands between the guard and the timer assignment; in that case it falls through to the idempotent Stop() instead of failing the egress. A genuinely frozen pipeline, EOS sent but never reaching the sinks, still fails after eosTimeout exactly as before.
The new controller test reproduces the scenario end-to-end with a real pipeline (silence generator -> opus -> ogg file sink): it sends EOS directly through the pipeline, invokes SendEOS the moment the bus EOS is processed, and asserts the egress completes with a finalized file. Without the guard it fails deterministically with the production signature, EGRESS_FAILED (error: "pipeline frozen").