Skip to content

compliance: scope edge-agentic checker (rename to edge_agentic.py, required ruleset/model args) #397

Description

@Palanivelg

Follow-up to PR #346 (thread #346 (comment)), flagged by @viraatc as fine to address in a follow-up.

compliance/checker.py is edge-agentic-specific (BFCL metric keys, qwen3.6-27b / mlperf-edge-current defaults, hardcoded attestation note) under a generic top-level compliance/ name.

Proposed:

  • Rename to compliance/edge_agentic.py (or move next to the edge ruleset).
  • Make ruleset_name/model_name required args instead of defaults.
  • Name the model's golden_accuracy keys to match the scorer's breakdown keys (overall_accuracy, normalized_single_turn_score) so the _ACCURACY_METRIC_KEYS mapping disappears.
  • Generalizing into a real framework can wait until a second benchmark needs one; coordinate with feat(compliance): MLPerf TEST04 caching audit — extensible AuditTest framework #332.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions