testmap

A feature × test-kind validation matrix for your test suite.

Running pytest --testmap on a project tagged with @testmap.

testmap answers a question code-coverage tools can't: what validation evidence do we have? Coverage tells you which lines ran. testmap tells you which kinds of test back each feature, and flags the required kinds that are missing.

Feature    Unit  Integration  Property  Perf  Status
----------------------------------------------------
parser        1            1         1     0       ✓
processor     1            1         0     0       ✗
sender        1            0         0     0       ✗

Missing:
  • processor: property
  • sender: integration

That output is useful for humans reviewing a PR, and for agents ("generate property tests for processor").

Highlights

• A matrix, not a percentage. Features down the side, test kinds across the top.
• Pluggable taxonomy. Define your own kinds (unit, integration, property, perf, …).
• Per-feature requirements. Set a global default, override it per feature.
• A pytest plugin that tags tests with a @testmap decorator and collects them at zero runtime cost.
• A standalone core + CLI that ingests the metadata and renders the report. No pytest required to read it.
• Agent-friendly JSON. --testmap-json emits machine-readable records for tooling and LLMs.

Installation

testmap ships as two distributions. Most users want the pytest plugin, which pulls in the core automatically:

pip install pytest-testmap
# or
uv add pytest-testmap

If you only need to render a report from a JSON file (e.g. in CI), install the standalone core on its own:

pip install testmap

Usage

1. Tag your tests

Mark each test with the feature it exercises and the kind of validation it provides:

from pytest_testmap import testmap

@testmap(feature="processor", kind="unit")
def test_processor_handles_empty_input():
    ...

@testmap(feature="processor", kind="integration")
def test_processor_round_trip():
    ...

2. Render the matrix

pytest --testmap

This prints the feature × kind matrix in the pytest terminal summary and flags any required kinds that are missing.

3. Report without running the suite

Add --testmap-only to render the matrix from collected metadata without executing a single test:

pytest --testmap-only

4. Report without pytest at all

The standalone testmap core can discover @testmap-tagged tests by scanning your source directly, so a report needs neither pytest nor a pre-generated file:

uvx testmap report          # scans testpaths from pyproject.toml
uvx testmap report src/     # or point it at a directory
testmap report --json       # machine-readable

5. Emit JSON for tooling

To split collection (in the test job) from reporting (in a separate step), have the plugin write the raw records and feed the file to the CLI later:

pytest --testmap-json testmap.json    # collect
testmap report testmap.json           # render that file

6. Gate commits and CI on coverage

testmap report exits non-zero whenever a feature is missing a required kind, so it doubles as a coverage gate with no extra flag:

testmap report && echo "all features covered"   # exits 1 if any are missing

A pre-commit hook that wires exactly that in lives in the testmap-pre-commit mirror repo (the same layout ruff-pre-commit uses). Reference it from your .pre-commit-config.yaml:

repos:
  - repo: https://github.com/tylerriccio33/testmap-pre-commit
    rev: v0.1.0   # a mirror tag
    hooks:
      - id: testmap

pre-commit installs that mirror (which pulls in testmap) and runs testmap report; a missing required kind fails the commit.

Configuration

The kind taxonomy and required kinds live under [tool.testmap] in pyproject.toml:

[tool.testmap]
kinds = ["unit", "integration", "property", "perf"]
required = ["unit", "integration"]   # global default

[tool.testmap.features]
processor = ["unit", "integration", "property"]   # per-feature override
docs = { exclude = ["perf"] }                     # opt a feature out of a kind

[tool.testmap.statuses]
complete = "✓"     # Status column symbol when a feature has every required kind
incomplete = "✗"   # ...and when it's missing one

required is the global default that every feature must satisfy; entries under [tool.testmap.features] override it for a specific feature. An entry can be a list of required kinds, or a table with exclude to drop kinds a feature doesn't need — those show as n/a in the matrix and never count as missing.

The [tool.testmap.statuses] symbols shown in the Status column default to ✓ / ✗; override either independently (the table is optional).

Generate features from source

By default a feature only exists once a test tags it — a feature with zero tests is invisible. [[tool.testmap.generate]] closes that gap: it derives the expected feature universe straight from your source, so anything with no tests shows up as an all-missing row instead of silently dropping out.

[[tool.testmap.generate]]
select = "functions"   # "functions" or "classes"
from = "src/**/*.py"    # glob (Linux rules, ** allowed), relative to pyproject.toml
where = "public"        # "public", "private", or "all" (default)

Each generator is a small select / from / where query. select picks the node kind, from globs the source files, and where filters by visibility (leading-underscore = private). Only top-level definitions count — the natural notion of a file's public API — and the feature name is the symbol name. List multiple [[tool.testmap.generate]] blocks to combine sets (e.g. public functions and public classes); the results are deduped. Generated features take the global required kinds unless a [tool.testmap.features] entry overrides them.

For example, given src/mod.py:

def parse(): ...
def _helper(): ...   # private, skipped
class Sender: ...

and this config:

[tool.testmap]
kinds = ["unit", "integration"]
required = ["unit"]

[[tool.testmap.generate]]
select = "functions"
from = "src/**/*.py"
where = "public"

[[tool.testmap.generate]]
select = "classes"
from = "src/**/*.py"
where = "public"

testmap report seeds the matrix with parse and Sender even though no test tags them yet, so they surface as uncovered instead of being invisible:

Feature  Unit  Integration  Status
----------------------------------
Sender      0            0       ✗
parse       0            0       ✗

Missing:
  • Sender: unit
  • parse: unit

Claude / agent skill

testmap ships a Claude Code skill that teaches an agent how to use the package — reading the report, tagging tests, selecting which functions to map, and writing each test kind. Install it into a project (or globally) with:

testmap install-skill          # → ./.claude/skills/testmap-do/
testmap install-skill --user   # → ~/.claude/skills/  (every project)

Agents can then act on requests like /testmap-do fill out the map for functions or /testmap-do skip private functions in this file.

Packages

This is a uv workspace monorepo with two distributions:

• testmap (packages/testmap): standalone core + CLI. Ingests test metadata and renders the matrix / missing report, exiting non-zero when a feature is under-covered.
• pytest-testmap (packages/pytest-testmap): pytest plugin. Provides the @testmap(feature=..., kind=...) decorator and the --testmap / --testmap-json options.

The pre-commit hook lives in a separate testmap-pre-commit mirror repo — see Gate commits and CI on coverage.

Development

uv sync          # set up the workspace venv
make lint        # ruff format + ruff check + pyrefly
make test        # pytest with coverage

Commits follow Conventional Commits. Install the git hooks (lint, type-check, commit-message check) with prek:

prek install --hook-type pre-commit --hook-type commit-msg

License

testmap is released under the MIT License.