Handle FPS mismatch between video source and video encoder output

[dimadesu]

Prompt

/grill-me i want lifestreamer app to try basing fps for setting off what video encoder supports, not what built-in camera supports. this app can have RTMP/SRT player (Exoplayer) as source - that can have different FPS than built-in camera as well as USB/UVC camera source - same. i need to have a way to switch between different video sources during one stream that will work when input fps is lower or higher than output video encoder FPS.

For inspiration you can check inspiration/moblin project. I think it has some timers for each frame and it either duplicates a previous frame or render black frame depending on the source if source FPS is lower than target. Not sure what happens when source FPS is higher then target.

---

Here is the complete list of files that have been changed across the project:

Main Application

• SettingsFragment.kt: Decoupled target FPS preferences from camera capabilities, showing values strictly based on encoder support.

StreamPack Submodule

• DefaultSurfaceProcessor.kt: Implemented the new timer-driven render loop running at the target FPS, duplicating or dropping frames as necessary, and handling the 500ms inactivity timeout (by sending black frames).
• ISurfaceProcessor.kt: Exposed setTargetFps in ISurfaceProcessorInternal.
• OpenGlRenderer.kt: Added the renderBlack method to clear EGL window buffers to black.
• VideoInput.kt: Updated processor instantiation and configuration flows to feed the target FPS parameter into the surface processor.

---

Implementation Plan - Decouple Video Source FPS from Video Encoder FPS

The goal is to decouple the video input source's framerate (e.g. built-in camera, USB/UVC camera, or RTMP/SRT ExoPlayer source) from the video encoder's target framerate. Currently, rendering and encoding are event-driven, triggered directly by the input source's onFrameAvailable callback. Under the new decoupled architecture, the renderer will run a stable, target FPS-driven render loop.

• When the input FPS is lower than the target FPS, the render loop will duplicate the previous frame.
• When the input FPS is higher than the target FPS, the render loop will naturally drop extra frames.
• When the source is stopped, disconnected, or switching (no frame received within a 500ms timeout), the render loop will render a solid black frame to keep the encoder fed.
• Settings will be updated so that the video FPS preference list is based on the video encoder's capabilities, not filtered by the camera. The warning about camera FPS support will be removed.

---

User Review Required

We have aligned on the following design decisions through the /grill-me session:

1. Frame Decoupling Behavior: Duplicate the last frame when the active source's frame rate is lower than target. Render a black frame when the source is stopped, disconnected, or switching.
2. Timeout Threshold: A 500ms timeout of no frames from the active input surface will declare it "disconnected/stopped" and transition the render loop from frame-duplication to black-frame rendering.
3. Camera FPS Warning: Remove the warning dialog in the Settings screen because the camera's FPS no longer restricts or blocks the target encoder FPS.

---

Proposed Changes

StreamPack Core Library

[MODIFY] ISurfaceProcessor.kt

• Add fun setTargetFps(fps: Int) to ISurfaceProcessorInternal to allow updating the target loop FPS.

[MODIFY] OpenGlRenderer.kt

• Add a renderBlack(timestampNs: Long, surface: Surface) method that clears the color buffer using GLES20.glClearColor(0.0f, 0.0f, 0.0f, 1.0f) and GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT) to draw a solid black frame on the specified output surface.

[MODIFY] DefaultSurfaceProcessor.kt

• Implement setTargetFps(fps: Int) on DefaultSurfaceProcessor.
• Add a periodic render loop (Runnable running via glHandler.postDelayed) that ticks at the target FPS. It should use a self-correcting timer to ensure precise and stable intervals.
• Track state: lastActiveSurfaceTexture: SurfaceTexture?, lastFrameTimeMs: Long, hasFrame: Boolean, and cached texture/output matrices.
• Modify onFrameAvailable(surfaceTexture) to only update the texture (surfaceTexture.updateTexImage()), store the matrix/timestamp, and set flags. Remove the direct rendering call.
• In the render loop tick:
  • If any output surface is streaming or previewing (i.e. surfaceOutput.isStreaming() is true):
    • Check if we have an active texture and the elapsed time since the last frame is $\leq$ 500ms.
    • If yes, draw the texture frame on the active surfaces with renderer.render using the current uptime timestamp (System.nanoTime()).
    • If no (timeout or no source), clear the active surfaces to solid black using renderer.renderBlack with the current uptime timestamp.
• Cancel the render loop inside release().

[MODIFY] VideoInput.kt

• In the constructor, call processor.setTargetFps(30) (or a sensible default).
• In applySourceConfig, call processor.setTargetFps(videoConfig.fps).
• In buildSurfaceProcessor, call newSurfaceProcessor.setTargetFps(videoSourceConfig.fps).

---

LifeStreamer App

[MODIFY] SettingsFragment.kt

• Update loadVideoSettings(encoder: String) where videoFpsListPreference is populated:
  • Instead of getting supported framerates via streamerInfo.video.getSupportedFramerates(..., cameraId), query streamerInfo.video.getSupportedFramerate(encoder) which returns the Range<Int> supported by the video encoder.
  • Filter FpsEntries (24, 25, 30, 60) based on whether they fall within this encoder range.
  • Remove the camera FPS warning from the setOnPreferenceChangeListener block.

---

Verification Plan

Automated Tests

• Run Gradle check tasks to ensure the core library and app modules build successfully:

  ./gradlew :StreamPack:core:assembleDebug
  ./gradlew :app:assembleDebug

Manual Verification

• Deploy the app to a device or emulator and run a stream.
• In Settings, verify that 60 FPS is selectable even if the built-in front camera only supports 30 FPS.
• Start streaming and switch video sources (e.g. from built-in Camera to UVC camera or RTMP player source). Verify that:
  • The stream continues to run at a stable target FPS.
  • During the source switch gap, a solid black frame is rendered, preventing connection drops or stream freezes.
  • Disconnecting a USB camera results in a solid black screen rather than a frozen last frame.