Skip to content

Add instant image translation (Google Lens style)#293

Open
quangdat99x wants to merge 1 commit into
ramjke:masterfrom
quangdat99x:feature/image-translation
Open

Add instant image translation (Google Lens style)#293
quangdat99x wants to merge 1 commit into
ramjke:masterfrom
quangdat99x:feature/image-translation

Conversation

@quangdat99x

Copy link
Copy Markdown

Summary

Adds a new Alt+D hotkey that turns a screen selection into a Google Lens style translation: OCR the region, auto-detect the source language, translate, and paint the translation over the original text positions in a floating overlay. A small toolbar offers source-override + target-language dropdowns (live re-translate), copy-all, and Esc/click-away to close. Default target is Vietnamese.

What''s inside

  • Positional Windows OCR + auto-detectTranslumo.OCR/WindowsOCR/WindowsOcrPositional.cs: keeps per-line bounding boxes and detects the source language by running the installed recognizer packs and scoring each result by how many characters match that language''s writing system (the UWP OCR API exposes no per-word confidence, so a script-match heuristic is used).
  • Auto-source translationAutoSourceGoogleTranslator sends sl=auto, reusing the proxy-aware GoogleContainer and result regex without disturbing the config-bound GoogleTranslator used by continuous translation.
  • OrchestrationImageTranslationService runs OCR then per-line translation with bounded concurrency.
  • OverlayImageTranslationOverlayWindow shows the frozen region snapshot with translated boxes at the original positions and the toolbar; changing a dropdown re-runs the pipeline and repaints.
  • Wiring — new ImageTranslateKey (default Alt+D) across HotKeysConfiguration / HotKeysServiceManager, a configurable row in Hotkeys settings, ChatWindowViewModel handler (region capture + overlay), DI registration, and localization strings (en/ru/zh).

Notes / limitations

  • Source auto-detect for the OCR step depends on which Windows OCR language packs are installed (Windows OCR has no native auto recognizer); the inline source dropdown lets the user override. The translation step always uses sl=auto.
  • Line-by-line translation preserves layout but loses cross-line context (same trade-off as Lens).

Test plan

  • dotnet build Translumo.sln -c Release — 0 errors.
  • Ran the app, pressed Alt+D, selected a region of English text: overlay auto-detected en-US, translated to Vietnamese over the original lines, and switching the target dropdown to Spanish live re-translated all lines. Copy and Esc work.

🤖 Generated with Claude Code

New Alt+D hotkey: select a screen region, then OCR it, auto-detect the
source language, translate, and paint the translation over the original
text positions in a floating overlay — with inline source/target language
dropdowns, copy, and Esc-to-close.

- Positional Windows OCR with per-line boxes + multi-pack language
  auto-detect scored by writing-system match (UWP OCR has no confidence
  API): Translumo.OCR/WindowsOCR/WindowsOcrPositional.cs
- AutoSourceGoogleTranslator (sl=auto) reusing the proxy-aware
  GoogleContainer, without touching the config-bound GoogleTranslator
- ImageTranslationService orchestrates OCR + bounded-concurrency
  per-line translation
- ImageTranslationOverlayWindow renders the frozen region snapshot with
  translated boxes and the toolbar; live re-translate on dropdown change
- Hotkey/DI wiring + localization (en/ru/zh); default target Vietnamese

Verified end to end: auto-detected en-US, translated to Vietnamese, and
live-switched target to Spanish over a real screen region.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant