perf(stream): share H264 capture frames & enhance H264(Webrtc) frame rate - #888
Conversation
Route Direct and WebRTC consumers through one H264 capture source to avoid duplicate VENC reads. Overlap WebRTC capture with ordered RTP/SRTP writes and packetize each access unit once for all WebRTC clients. Add libopencv_video.so.409 into dl_lib.
|
Sharing one capture between the two H.264 paths is the right idea. Three things I would look at before this lands. A slow WebRTC peer stalls the direct viewers. A non-blocking hand-off fixes it: a one-deep slot per subscriber that either refuses a frame while one is pending or replaces it, and a counter for what was dropped. The subscriber that fell behind then needs a keyframe, which the source already knows about from
The entry comes from MaixCDK: its I understand why the file is here: the cross-linker reports every dependency as The frame rate gain is partly loss repair being switched off. For context, |
|
@yuzi-co Your response has been very helpful! I’ll take your advice and submit more suitable code later. |
Keep slow WebRTC consumers from blocking the shared H264 source by dropping buffered frames until the next keyframe. Restore the WebRTC NACK and RTCP report interceptors while retaining a safe RTP MTU, and enable linker --as-needed for libkvm to remove unused OpenCV dependencies.
Direct mode and WebRTC mode each ran their own ticker and each called ReadH264. There is one encoder behind that call and it hands whichever caller asks whatever frame is ready, so two loops do not each receive the stream: they divide it. A viewer on direct mode and a viewer on WebRTC at the same time therefore each got roughly every second frame of a GOP, which decodes to nothing useful, and the board paid for the capture twice to produce it. The frame rate counter was updated by both loops as well, so the figure reported was double the frames captured. stream.H264Source is one capture loop for both. It starts with the first subscriber and stops after the last one leaves, and it owns the ticker, the screen snapshot and the FPS changes that the two loops each kept a copy of. The hand-off is a FrameSlot, not a channel the producer waits on. The capture loop must never block on a delivery path: the two paths share it now, so a WebRTC writer waiting on a slow peer would otherwise stop the direct viewers as well. A path that is not ready has the frame refused and counted, which is the policy FrameSlot already documents for H.264 and the same one each path uses for its own clients. Demand is asked before the encoder is read, because a read costs the board whether or not anyone takes the result. Direct mode acknowledges frames and stops asking while a viewer is behind, and that gate survives: the source reads nothing while every subscriber is quiet. Each path still reports capture status under its own mode, so a failed read is delivered rather than swallowed. A status nobody reports is a stream that fails with nothing said. The delivery loops now wait on frames rather than on a ticker, so they need their own way to notice the last viewer leaving. A one second idle tick does it. Nothing is being served in the meantime, and stopIfIdle rechecks the client count under the mutex, so a viewer arriving inside that second is not stranded. The idea is upstream PR sipeed#888. Its own hand-off blocks the capture loop on a full four-deep channel, with a WebRTC track write behind that, so one slow peer stalls the direct viewers too. That part is not taken.
3 participants
Summary
libkvm.so.Details
The Direct and WebRTC streaming paths previously read H.264 frames independently from the same encoder queue. When both modes were active, consumers could compete for frames and receive incomplete streams.
This change routes both paths through a shared H.264 source so the encoder is read only once per frame.
Each H.264 subscription now uses a bounded non-blocking hand-off. If a subscriber falls behind and its queue overflows, buffered frames are discarded and that subscriber waits for the next keyframe before receiving P-frames again. This prevents a slow WebRTC consumer from applying backpressure to the shared source and interrupting Direct viewers.
The WebRTC interceptor configuration is kept intentionally narrow:
The RTP packetization MTU is set to
1200to remain safe across IPv6, TURN, VPN, and other paths with reduced MTU.The
libkvm.solink now uses--as-needed. This removes unused OpenCV dependencies from the final ELF, includinglibopencv_video.so.409, without affecting the OpenCV symbols used by the library.