Gap
The repository has no unit-test framework wired into CI. Today:
- There is no
test task in mise.toml, and mise run build (the CI gate) runs lint / fmt / validate / security scanners — but no tests.
pytest (or any test runner) is not declared as a tool in mise.toml or .pre-commit-config.yaml.
- Exactly one test-shaped file exists —
tools/evals/databases-on-aws/dsql/scripts/test_safe_query.py — and it is not invoked by mise or any workflow. It's a standalone/manual file.
Net effect: Python logic in tools/ (and plugin scripts) can regress silently. Contributors have no standard place to put tests, and no assurance they run.
Proposal
Establish a lightweight Python unit-test convention and gate it in CI:
- Add a test runner. Add
pytest to mise.toml [tools] (e.g. "pipx:pytest" or "npm:"-equivalent — pick per repo tooling norms). (Adding a dependency — needs maintainer sign-off per the repo's "no new deps without asking" rule.)
- Add a
mise run test task that discovers and runs tests (e.g. pytest tools/ plugins/ or a curated set of paths).
- Wire it into
mise run build so CI fails on a broken test — add { task = "test" } to the build chain in mise.toml.
- Document the convention (test file location + naming, e.g.
test_*.py next to the code or under a tests/ dir) in docs/DEVELOPMENT_GUIDE.md.
- Handle bandit B101. Bandit runs with defaults (no config), so
assert in tests trips B101. Either add a bandit config that skips B101 for test paths, or follow the existing precedent of per-line # nosec B101 - test assertion markers (see test_safe_query.py, which carries 21 of them). A shared config is cleaner than per-line markers at scale.
First example to add: guard the vendored-Semgrep parser (from #220)
Once the framework lands, add a test for tools/semgrep/rules-update.py (the vendored-ruleset updater introduced in #220). Its text-based split_rules() / rule_id_of() parser has a documented assumption worth guarding: a rule's own content never begins a column-0 - line (that marker is what delimits rules). A reviewer flagged a fixture test as a good follow-up.
The test (already prototyped and passing during #220 review, then deferred to keep that PR scoped) should cover:
split_rules returns header + all blocks and is byte-preserving (header + "".join(blocks) == input) — the property that keeps retained rules identical to the upstream download.
- id extraction works when
id: is not on the first line — ~14 rules in r/all lead with - patterns: / - fix: and carry id: on a later line.
- indented
- inside a message/fix block scalar does NOT cause a spurious split (the positive side of the assumption).
- a column-0
- in content fails loud — a mis-split fragment has no id:, so rule_id_of raises UpdateError (loud abort, not silent corruption). This is the fail-safe that makes the assumption tolerable.
ID_RE does not match nested keys (rule_id, r_id, rv_id, version_id).
split_rules raises on a rule-less document (rules: []).
Implementation notes for that test:
rules-update.py has a hyphen, so import it via importlib.util.spec_from_file_location rather than a plain import.
- A zero-dependency
__main__ runner + pytest-shim (as in test_safe_query.py) lets it run via uv run tools/semgrep/test_rules_update.py even before the framework lands.
Acceptance criteria
Follow-up to #220 (do not block that PR on this).
Filed with assistance from Claude Code.
Gap
The repository has no unit-test framework wired into CI. Today:
testtask inmise.toml, andmise run build(the CI gate) runs lint / fmt / validate / security scanners — but no tests.pytest(or any test runner) is not declared as a tool inmise.tomlor.pre-commit-config.yaml.tools/evals/databases-on-aws/dsql/scripts/test_safe_query.py— and it is not invoked bymiseor any workflow. It's a standalone/manual file.Net effect: Python logic in
tools/(and plugin scripts) can regress silently. Contributors have no standard place to put tests, and no assurance they run.Proposal
Establish a lightweight Python unit-test convention and gate it in CI:
pytesttomise.toml[tools](e.g."pipx:pytest"or"npm:"-equivalent — pick per repo tooling norms). (Adding a dependency — needs maintainer sign-off per the repo's "no new deps without asking" rule.)mise run testtask that discovers and runs tests (e.g.pytest tools/ plugins/or a curated set of paths).mise run buildso CI fails on a broken test — add{ task = "test" }to thebuildchain inmise.toml.test_*.pynext to the code or under atests/dir) indocs/DEVELOPMENT_GUIDE.md.assertin tests tripsB101. Either add a bandit config that skips B101 for test paths, or follow the existing precedent of per-line# nosec B101 - test assertionmarkers (seetest_safe_query.py, which carries 21 of them). A shared config is cleaner than per-line markers at scale.First example to add: guard the vendored-Semgrep parser (from #220)
Once the framework lands, add a test for
tools/semgrep/rules-update.py(the vendored-ruleset updater introduced in #220). Its text-basedsplit_rules()/rule_id_of()parser has a documented assumption worth guarding: a rule's own content never begins a column-0-line (that marker is what delimits rules). A reviewer flagged a fixture test as a good follow-up.The test (already prototyped and passing during #220 review, then deferred to keep that PR scoped) should cover:
split_rulesreturns header + all blocks and is byte-preserving (header + "".join(blocks) == input) — the property that keeps retained rules identical to the upstream download.id:is not on the first line — ~14 rules inr/alllead with- patterns:/- fix:and carryid:on a later line.-inside amessage/fixblock scalar does NOT cause a spurious split (the positive side of the assumption).-in content fails loud — a mis-split fragment has noid:, sorule_id_ofraisesUpdateError(loud abort, not silent corruption). This is the fail-safe that makes the assumption tolerable.ID_REdoes not match nested keys (rule_id,r_id,rv_id,version_id).split_rulesraises on a rule-less document (rules: []).Implementation notes for that test:
rules-update.pyhas a hyphen, so import it viaimportlib.util.spec_from_file_locationrather than a plainimport.__main__runner + pytest-shim (as intest_safe_query.py) lets it run viauv run tools/semgrep/test_rules_update.pyeven before the framework lands.Acceptance criteria
mise run testexists and runs the Python test suite.mise run buildfails if any test fails.docs/DEVELOPMENT_GUIDE.md.rules-update.pyparser test above is included as the first example.Follow-up to #220 (do not block that PR on this).
Filed with assistance from Claude Code.