feat: dynamically fetch and embed AI models list

[abhinavgautam01]

Part of #42

Description

This PR adds an opt-in generated AI model catalogue for tool mention detection using OpenRouter model data.

The catalogue is disabled by default and can be enabled with --include-model-catalog. This keeps the default detector path unchanged while allowing users to scan for a broader set of model names when they explicitly opt in.

Broader design questions, including Hugging Face coverage and custom user-provided model lists, remain open for discussion in #42.

Changes Made

• Added detection/toolmention/generate.go to fetch model data from the OpenRouter API.
• Added fixture-tested model generation logic under detection/toolmention/modelgen.
• Generated detection/models.go with source metadata, normalized model count, and model snapshot hash.
• Added opt-in model catalogue support through --include-model-catalog.
• Added ambiguity filtering for common prose collisions.
• Added longest-match suppression so overlapping mentions prefer the most specific match.
• Added regression tests for ambiguous model names and generated catalogue behavior.
• Added no-match benchmarks for default and opt-in catalogue detection paths.

Testing

• Ran go generate ./...
• Ran gofmt
• Ran go mod tidy
• Ran git diff --check
• Ran go test ./...
• Ran go build -v ./...
• Ran go test -bench=BenchmarkDetectNoMatch ./detection/toolmention -run '^$'

[andrew]

Please rebase and integrate the generated models with the current detection.SupportedToolsInMentions structure. detection/toolmention/toolmention.go currently conflicts with main.

Please filter generic OpenRouter identifiers before adding them. Entries such as auto, free, and router match ordinary prose; for example, “Use the free router setting” reports two AI signals and exits with status 1. Add negative tests for common words.

Please also deduplicate models case-insensitively and choose one canonical display value. A single GPT-4 mention currently produces separate GPT-4 and gpt-4 findings.

Finally, validate a successful HTTP status and a non-empty response before creating models.go. A JSON error response currently parses as an empty list and overwrites the generated file.

[abhinavgautam01]

Thanks, addressed these

• Rebasing onto current main resolved the toolmention.go conflict.
• Generated model mentions now integrate through detection.SupportedToolsInMentions.
• Filtered generic OpenRouter router entries before adding them.
• Added case-insensitive dedupe so mentions like gpt-4 produce the canonical GPT-4 finding once.
• Hardened generation to validate successful HTTP status, non-empty response body and non-empty model data before writing models.go.
• Added negative tests for common prose like “Use the free router setting”.

[omkar-foss]

We should include models from Hugging Face API along with the current OpenRouter API ones in here.

Also the list being generated into models.go file in here will bloat up the disclosure binary, so we need to discuss and do this another way.

May be we could simplify this by giving more control to user to pass their own custom models list as a cli argument? I guess this let's discuss on parent issue #42 and get the feature finalized before implementation.

[andrew]

The generated list still includes ambiguous entries such as Command R, R1, Sonar, and Spotlight. Ordinary technical prose about a keyboard shortcut, resistor, sonar data, or UI spotlight produces an AI finding and exit status 1. Please apply a broader ambiguity filter or require model-related context, and add negative tests for these cases.

[abhinavgautam01]

Thanks @andrew fixed. I added a broader ambiguity filter for Command R, R1, Sonar and Spotlight, removed those entries from the checked-in generated model list and updated the generator so they are not reintroduced. I also added detector-level negative tests for the prose examples and a CLI regression test confirming disclosure text exits with no AI signal for those ordinary phrases.

[abhinavgautam01]

We should include models from Hugging Face API along with the current OpenRouter API ones in here.

Also the list being generated into models.go file in here will bloat up the disclosure binary, so we need to discuss and do this another way.

May be we could simplify this by giving more control to user to pass their own custom models list as a cli argument? I guess this let's discuss on parent issue #42 and get the feature finalized before implementation.

Thanks @omkar-foss, agreed. I did not try to redesign that part in this PR. The Hugging Face source, binary-size concern from embedding models.go and possible custom model-list CLI input should be finalized on issue #42 before changing the implementation direction.

[andrew]

The ambiguity filter remains too narrow for a live model catalogue. “A body builder updated the training plan” detects Body Builder, and “The weaver repaired the fabric” detects Weaver. Please use contextual matching or restrict the generated data to identifiers suitable for prose scanning, with negative tests.

Prefix overlaps also produce several findings for one mention. Claude Opus 4.6 returns Claude, Claude Opus 4, and Claude Opus 4.6; GPT-4 Turbo returns both GPT-4 and GPT-4 Turbo. Please prefer the longest match and add regression tests.

This raises the detector from 22 to 654 regexes, all checked against every message. Please make the generated catalogue opt-in or add detector selection and a no-match benchmark before enabling it by default.

Running go generate immediately changed 257 lines in models.go. Please add fixture-based generator tests and record enough source metadata to identify the OpenRouter snapshot represented by the checked-in list.

[abhinavgautam01]

Thanks, fixed. The generated model catalogue is now opt-in through --include-model-catalog instead of being enabled by default, so the default detector path stays on the curated mention list.

I also updated toolmention matching to prefer the longest non-overlapping match, added regressions for Claude Opus 4.6 and GPT-4 Turbo, filtered the ambiguous generated entries called out in review and added negative tests for the ordinary prose cases including body builder/weaver examples.

For the generator, I moved parsing/rendering into a fixture-tested helper package and added snapshot metadata to models.go so the checked-in OpenRouter-derived list records its source, normalized count and normalized SHA256. I also added no-match benchmarks for default and catalogue-enabled matching.

[andrew]

Thanks, this is close. A few things before it can land:

Rebase needed. The branch now conflicts with main (cmd/cmd.go changed in #73, and #57 touched the toolmention/constants area).

Regenerate models.go. The checked-in file header has Normalized model count and Normalized model SHA256 but not the Source item count line that RenderModelsGo now emits, so it wasn't regenerated after the renderer change. Running go generate ./... will immediately dirty it.

Ambiguity filter misses slug forms. The filter blocks "body builder" but the checked-in list still contains "bodybuilder", and TestBuildSnapshotFromFixture asserts that "body-builder" survives normalisation. Please filter the slug/hyphenated variants too. While there, "Uncensored", "Command A", "Saba", "o1", and "o3" are also common-prose collisions worth adding. Since the catalogue is opt-in the blast radius is small, but the fixture test currently documents the opposite of the intent.

Dead check in Detect(). The first candidate-collection loop tests if seen[seenKey] { continue } but nothing writes to seen until after it's reassigned below, so the guard never fires. Drop it (or drop the reassignment and let one map serve both loops).

The opt-in flag, longest-match suppression, lazy sync.Once compilation, and benchmarks all look good. Note the longest-match change also alters default-detector output ("Generated with Claude Code" now yields only Claude Code, not Claude Code + Claude), which I think is an improvement but worth being aware of.

On @omkar-foss's point about settling the design on #42 first: the ~660 embedded strings add roughly 15KB to a binary that already links go-git, so I'm not worried about size, and opt-in keeps the default path unchanged. I'd be fine landing this as an interim, but I'll leave it to the two of you to agree on #42 before I approve.

[abhinavgautam01]

Thanks Andrew, addressed these.

Rebased the branch onto current main and resolved the tool mention conflict with the updated SupportedToolMentions structure.

Regenerated models.go; it now includes the Source item count metadata from RenderModelsGo.

The ambiguity filter now uses a normalized key, so spaced/hyphenated/concatenated variants are filtered consistently. Added coverage for body builder/body-builder/bodybuilder, Uncensored, Command A, Saba, o1 and o3 in both generator and detector/CLI tests.

Also removed the dead seen check from the candidate collection loop in Detect() while keeping the opt-in catalogue, lazy sync.Once compilation and longest-match suppression.

[abhinavgautam01]

hey @omkar-foss, that makes sense.

I’ve updated this PR so the generated OpenRouter model catalogue is opt-in via --include-model-catalog, so the default detector path and binary behavior stay unchanged unless the user explicitly enables it. Andrew also noted the embedded list adds roughly 15KB, so binary size should be acceptable for this interim implementation.

i agree the longer-term design, including Hugging Face coverage and/or letting users provide a custom model list, should be discussed on issue #42 before expanding this further. For this PR, I’d like to keep the scope to an opt-in generated catalogue from OpenRouter with ambiguity filtering and tests, then follow up on #42 for the broader design...

What do you think..!?

[omkar-foss]

hey @omkar-foss, that makes sense.

I’ve updated this PR so the generated OpenRouter model catalogue is opt-in via --include-model-catalog, so the default detector path and binary behavior stay unchanged unless the user explicitly enables it. Andrew also noted the embedded list adds roughly 15KB, so binary size should be acceptable for this interim implementation.

i agree the longer-term design, including Hugging Face coverage and/or letting users provide a custom model list, should be discussed on issue #42 before expanding this further. For this PR, I’d like to keep the scope to an opt-in generated catalogue from OpenRouter with ambiguity filtering and tests, then follow up on #42 for the broader design...

What do you think..!?

Hi Abhinav, thanks for your response. I think for an interim solution the diff in this PR is a bit large, as an entire package is being generated (modelgen) which could instead be a json file with the models list fetched from openrouter or huggingface.

Regarding my previous comment of bloat, which @andrew mentioned above (~660 embedded strings add roughly 15KB), the binary size is still fine but I'm worried about the time complexity of 660 additional entries when they're matched against a repo with 100s of commits. And the list could grow even larger. Also the newly added --include-model-catalog flag belongs to a separate issue #58, which is also something that needs discussion either on that issue or on the slack channel.

Considering the above, I'd recommend you and @andrew to kindly keep further code changes to this PR on hold until things have been discussed properly either on respective issues or on slack. My main concern is if add things without proper deliberation, it'll increase the difficulty to maintain this project.

[abhinavgautam01]

Thanks Omkar, that’s fair.

I’ll hold off on further code changes to this PR until the model-catalog approach is discussed properly on #42 / #58 or Slack.

Just to clarify one point: modelgen is the generator helper package, while detection/models.go is the generated catalogue. But I agree the current PR is still a larger implementation than an interim change should probably be without design agreement.

I’ve already updated the PR wording from fixes #42 to Part of #42 so it won’t imply that the broader issue is resolved. I’m fine with reworking this after everyone agrees on the preferred direction, whether that’s a JSON model list, Hugging Face/OpenRouter source handling, user-provided model files, detector selection or something else.