Skip to content

Establish a Python unit-test framework wired into mise run build #226

Description

@scottschreckengaust

Gap

The repository has no unit-test framework wired into CI. Today:

  • There is no test task in mise.toml, and mise run build (the CI gate) runs lint / fmt / validate / security scanners — but no tests.
  • pytest (or any test runner) is not declared as a tool in mise.toml or .pre-commit-config.yaml.
  • Exactly one test-shaped file exists — tools/evals/databases-on-aws/dsql/scripts/test_safe_query.py — and it is not invoked by mise or any workflow. It's a standalone/manual file.

Net effect: Python logic in tools/ (and plugin scripts) can regress silently. Contributors have no standard place to put tests, and no assurance they run.

Proposal

Establish a lightweight Python unit-test convention and gate it in CI:

  1. Add a test runner. Add pytest to mise.toml [tools] (e.g. "pipx:pytest" or "npm:"-equivalent — pick per repo tooling norms). (Adding a dependency — needs maintainer sign-off per the repo's "no new deps without asking" rule.)
  2. Add a mise run test task that discovers and runs tests (e.g. pytest tools/ plugins/ or a curated set of paths).
  3. Wire it into mise run build so CI fails on a broken test — add { task = "test" } to the build chain in mise.toml.
  4. Document the convention (test file location + naming, e.g. test_*.py next to the code or under a tests/ dir) in docs/DEVELOPMENT_GUIDE.md.
  5. Handle bandit B101. Bandit runs with defaults (no config), so assert in tests trips B101. Either add a bandit config that skips B101 for test paths, or follow the existing precedent of per-line # nosec B101 - test assertion markers (see test_safe_query.py, which carries 21 of them). A shared config is cleaner than per-line markers at scale.

First example to add: guard the vendored-Semgrep parser (from #220)

Once the framework lands, add a test for tools/semgrep/rules-update.py (the vendored-ruleset updater introduced in #220). Its text-based split_rules() / rule_id_of() parser has a documented assumption worth guarding: a rule's own content never begins a column-0 - line (that marker is what delimits rules). A reviewer flagged a fixture test as a good follow-up.

The test (already prototyped and passing during #220 review, then deferred to keep that PR scoped) should cover:

  • split_rules returns header + all blocks and is byte-preserving (header + "".join(blocks) == input) — the property that keeps retained rules identical to the upstream download.
  • id extraction works when id: is not on the first line — ~14 rules in r/all lead with - patterns: / - fix: and carry id: on a later line.
  • indented - inside a message/fix block scalar does NOT cause a spurious split (the positive side of the assumption).
  • a column-0 - in content fails loud — a mis-split fragment has no id:, so rule_id_of raises UpdateError (loud abort, not silent corruption). This is the fail-safe that makes the assumption tolerable.
  • ID_RE does not match nested keys (rule_id, r_id, rv_id, version_id).
  • split_rules raises on a rule-less document (rules: []).

Implementation notes for that test:

  • rules-update.py has a hyphen, so import it via importlib.util.spec_from_file_location rather than a plain import.
  • A zero-dependency __main__ runner + pytest-shim (as in test_safe_query.py) lets it run via uv run tools/semgrep/test_rules_update.py even before the framework lands.

Acceptance criteria

  • mise run test exists and runs the Python test suite.
  • mise run build fails if any test fails.
  • Test convention documented in docs/DEVELOPMENT_GUIDE.md.
  • Bandit B101-in-tests handled repo-wide (config or documented marker convention).
  • The rules-update.py parser test above is included as the first example.

Follow-up to #220 (do not block that PR on this).

Filed with assistance from Claude Code.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions