Add MARENGO search mode using TwelveLabs embeddings#9
Open
mohit-twelvelabs wants to merge 1 commit into
Open
Conversation
Adds an opt-in MARENGO mode that ranks captions by cosine similarity of TwelveLabs Marengo (512-dim) text embeddings, alongside the existing FUZZY and SEMANTIC modes. The TwelveLabs client is created lazily so the twelvelabs package and TWELVELABS_API_KEY are only required when MARENGO is selected; FUZZY/SEMANTIC behaviour is unchanged. Wires the mode into the CLI choices and the web GUI dropdown, documents it in the README, lists twelvelabs as an optional requirement, and adds tests (offline cosine-ranking test plus a live test skipped without a key).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
This PR adds an opt-in MARENGO search mode to Scoper, alongside the existing
FUZZYandSEMANTICmodes.What it adds
MARENGOmode that ranks a video's captions by cosine similarity of TwelveLabs Marengo multimodal embeddings (512-dim) — both the query and each caption are embedded into a shared space and ranked.get_timestampdispatch, the CLI--modechoices (MARENGO/M), and the web GUI algorithm dropdown.twelvelabsentry inrequirements.txt.test_marengo.py: an offline cosine-ranking test (network stubbed) and a live 512-dim embedding test that is skipped unlessTWELVELABS_API_KEYis set.Why it helps
The existing
SEMANTICmode trains Word2Vec on the Brown corpus, so it only "knows" words in that vocabulary and needs a custom dataset for anything else. Marengo embeds the full query/caption text with no model to train, handling out-of-vocabulary words, phrasing, and intent much better — a natural upgrade for the project's semantic-search goal.Opt-in / non-breaking
The default mode stays
FUZZY, andSEMANTICis untouched. The TwelveLabs client is created lazily on first use, so neither thetwelvelabspackage nor an API key is required unless someone explicitly selectsMARENGO. Existing behaviour and dependencies are unchanged.How it was tested
pytest test_marengo.py— offline ranking test passes; live test passes against the real API (confirmed a real text embedding is a 512-dim vector). With no key set, the live test skips cleanly.pyflakesis clean on the changed files.You can grab a free API key at https://twelvelabs.io — there's a generous free tier.