feat: six AI features across two phases — dubbing, bg removal, B-roll - #4
feat: six AI features across two phases — dubbing, bg removal, B-roll#4Ekaanth wants to merge 2 commits into
Conversation
…, captions, speakers, multicam sync Phase 1 (new AI capabilities, privacy-first): - AI Dubbing (local engine upgrade): new NLLB-200 translate-service (services/translate-service, port 8427) gives the existing dubbing flow a fully-local translation path. The "local" engine now batch-translates via NLLB with LLM fallback. Also adds a full-pipeline alternative at /api/dub/* (dubbing_service orchestrates whisper → translate → XTTS → ffmpeg). - Video Background Removal: per-frame rembg matting via a new /remove-bg-video endpoint on image-service (in-process job store + status polling). Proxied through ai-backend /api/background/*. Result inserted non-destructively as an alpha WebM above the source clip. - Auto B-roll from your footage: matches each transcript segment to the existing CLIP embedding index (IndexedDB) and auto-inserts the best clip. New /api/broll/* backend route. Reuses the existing visual-search index — no parallel store. Phase 2 (workflow features, reuse-first — zero new services, zero new deps): - Multilingual captions: existing translate flow upgraded to NLLB-first with multi-language quick-add chips and per-track SRT/VTT export via the existing subtitle endpoint (now accepts an optional filename). - Edit by speaker: scope operations (remove / tighten gaps / isolate) to a single speaker using diarization. Cut/compact helpers extracted from Smart Cut into lib/timeline-edits.ts (shared, no behavior change). - Auto multicam sync: client-side audio cross-correlation aligns camera angles without timecode. Pick a reference, preview offsets with scores, apply in one undoable transaction. Pure Web Audio, no backend. Infra: job_queue.py parameterized per-domain Redis prefix (dub_job: / vbg_job: singletons) so jobs don't collide with YouTube's. translate-service added to docker-compose + gpu override. README + roadmap updated. Verification: biome clean on all 22 changed/new frontend files; python3 -m py_compile clean on all changed backend files.
![vorflux[bot]](https://avatars.githubusercontent.com/in/2483280?s=60&v=4){kind=small}
There was a problem hiding this comment.
Security Review
Overall the code quality is solid — path-traversal guards are present, file extensions are validated on most endpoints, job IDs use UUIDs, no secrets or API keys are hardcoded, and FFmpeg commands are properly constructed with asyncio.create_subprocess_exec(*cmd) (list-based, no shell injection risk). The architecture of proxying through the ai-backend rather than exposing internal services directly to the browser is a good pattern.
Found a few hardening opportunities — none critical given this is a local-first Docker app, but worth addressing:
Summary of findings
- No file size limits on upload endpoints (background, dub, broll, image-service) — disk/memory exhaustion risk
- Missing extension allowlist in
background.pyproxy — shouldn't forward unsupported files to the downstream service - Internal service URLs leaked in 503 error messages — reveals infrastructure topology
- Permissive CORS on translate-service —
allow_credentials=Truewithallow_originslimited to localhost is acceptable for a local service, but worth being explicit about
See inline comments for details on each.
|
|
||
| @router.post("/remove-video") | ||
| async def remove_video_background( | ||
| file: UploadFile = File(...), |
There was a problem hiding this comment.
Missing file extension allowlist. Unlike dub.py (line ~82) and broll.py (line ~111), this proxy endpoint does not validate the uploaded file's extension before forwarding to the image-service. While the image-service does its own validation, validating early here avoids forwarding unsupported files unnecessarily and provides a clearer error to the caller.
| def _service_down(url: str) -> HTTPException: | ||
| return HTTPException( | ||
| status_code=503, | ||
| detail=( |
There was a problem hiding this comment.
Info leakage: internal service URL exposed. The _service_down helper (background.py line 26-33, broll.py line 41, translate.py line 85, dub.py line 51) returns the internal service URL directly to the caller in a 503 detail message. For a local Docker app this is low severity, but it reveals infrastructure topology. Consider logging the URL on the server side and returning a generic 'service unavailable' message.
| """Stream a produced transparent WebM back to the browser. | ||
|
|
||
| The image-service owns the file; we proxy the bytes so the browser only | ||
| needs the ai-backend origin. |
There was a problem hiding this comment.
Good: explicit path-traversal guard here (/, \\, ..). Consider adding this same guard pattern to the filename field in transcribe.py line 99, which accepts arbitrary user input and echoes it back in the response as a download suggestion.
|
|
||
| @router.post("/create") | ||
| async def create_dub( | ||
| file: UploadFile = File(...), |
There was a problem hiding this comment.
No file size limit on upload. Reads the entire file into memory via await file.read() (line ~90) with no size check. A multi-GB video could exhaust memory or crash the backend. Same applies to broll.py line ~120, background.py line ~46, and image-service/app.py line ~610. Consider checking file.size (if available from the client) or limiting at the middleware/Starlette level with Request(max_size=...).
| default=None, | ||
| description="Optional style settings for ASS format (font_name, font_size, primary_color, etc.)", | ||
| ) | ||
| filename: str | None = Field( |
There was a problem hiding this comment.
Unvalidated user input echoed in response. The new filename field accepts arbitrary strings and is returned verbatim in the JSON response (line ~136). The client uses this as a download filename, so it's low risk, but consider sanitizing it — at minimum strip path separators (/, \\) to prevent any confusion if a future client uses it differently.
| "http://localhost:3100", | ||
| "http://localhost:5173", | ||
| ], | ||
| allow_credentials=True, |
There was a problem hiding this comment.
Permissive CORS configuration. allow_credentials=True with allow_methods=["*"] and allow_headers=["*"] works for a local-only service but is a broad grant. Consider narrowing allow_headers to only the headers the frontend actually sends, and adding an explicit expose_headers list.
…sts, generic 503 messages, filename sanitization, and narrowed CORS - Add file size checks (HTTP 413, 500MB) to all upload endpoints: background.py, dub.py (main file + speaker_wav), broll.py, transcribe.py _proxy_file_upload, image-service (remove_bg + remove_bg_video) - Add video extension allowlist (ALLOWED_VIDEO_EXTS) to background.py proxy endpoint - Replace internal service URLs in 503 detail messages with server-side logging + generic user-facing messages across all routes (background, broll, dub, translate, transcribe) - Strip path separators from user-supplied filename in transcribe.py subtitle generation - Narrow translate-service CORS: explicit methods + headers instead of catch-all wildcards
There was a problem hiding this comment.
Security Review — All Issues Resolved ✅
Commit 8f989a0 addresses all 6 findings from the initial review, plus one additional URL leak caught during follow-up:
| # | Finding | Resolution | File(s) |
|---|---|---|---|
| 1 | No file size limits | HTTP 413 checks against 500MB MAX_UPLOAD_SIZE |
background.py, dub.py, broll.py, transcribe.py, image-service/app.py |
| 2 | Missing extension allowlist | ALLOWED_VIDEO_EXTS validated in proxy before forwarding |
background.py |
| 3 | URL leakage in 503 messages | logger.error(...) server-side, generic detail to client |
background.py, broll.py, dub.py, translate.py, transcribe.py |
| 4 | Unsanitized filename | Path separators (/, \) stripped |
transcribe.py |
| 5 | Permissive CORS | Narrowed to GET,POST,OPTIONS / Content-Type,Authorization |
translate-service/app.py |
| — | Whisper URL in 503 (caught during re-review) | Logged instead of leaked | transcribe.py |
All 7 files pass python3 -m py_compile. The changes follow existing patterns in the codebase (audio.py, engagement.py). No blocking issues remain.
Verdict: Approve
1 participant
Phase 1 (new AI capabilities, privacy-first):
AI Dubbing (local engine upgrade): new NLLB-200 translate-service (services/translate-service, port 8427) gives the existing dubbing flow a fully-local translation path. The "local" engine now batch-translates via NLLB with LLM fallback. Also adds a full-pipeline alternative at /api/dub/* (dubbing_service orchestrates whisper → translate → XTTS → ffmpeg).
Video Background Removal: per-frame rembg matting via a new /remove-bg-video endpoint on image-service (in-process job store + status polling). Proxied through ai-backend /api/background/*. Result inserted non-destructively as an alpha WebM above the source clip.
Auto B-roll from your footage: matches each transcript segment to the existing CLIP embedding index (IndexedDB) and auto-inserts the best clip. New /api/broll/* backend route. Reuses the existing visual-search index — no parallel store.
Phase 2 (workflow features, reuse-first — zero new services, zero new deps):
Multilingual captions: existing translate flow upgraded to NLLB-first with multi-language quick-add chips and per-track SRT/VTT export via the existing subtitle endpoint (now accepts an optional filename).
Edit by speaker: scope operations (remove / tighten gaps / isolate) to a single speaker using diarization. Cut/compact helpers extracted from Smart Cut into lib/timeline-edits.ts (shared, no behavior change).
Auto multicam sync: client-side audio cross-correlation aligns camera angles without timecode. Pick a reference, preview offsets with scores, apply in one undoable transaction. Pure Web Audio, no backend.
Infra: job_queue.py parameterized per-domain Redis prefix (dub_job: / vbg_job: singletons) so jobs don't collide with YouTube's. translate-service added to docker-compose + gpu override. README + roadmap updated.
Verification: biome clean on all 22 changed/new frontend files; python3 -m py_compile clean on all changed backend files.
We are not currently accepting PRs except for critical bugs.
If this is a bug fix:
If this is a feature:
This PR will be closed. Please open an issue to discuss first.
Session Details
(aside)to your comment to have me ignore it.