ci: block a Harbor task that gives away its own answer

.github/workflows/validate.yml

@@ -46,6 +46,32 @@ jobs:
       - name: Eval schema
         run: python3 tools/run_evals.py --validate
 
+      # First, because a gate is worth what the detector behind it is worth and every way
+      # this one can break is silent: a regex that stops matching reports a clean zero for
+      # every task, which reads exactly like a repository whose tasks are all sound. So it
+      # asserts itself against the skills and tasks in this tree — that a dotted call, a
+      # `::` call and a keyword argument are all still seen, that an alias bound twice
+      # keeps both meanings, that one more given-away symbol costs one point and a pasted
+      # line costs three, and that the budget below is still the worst task's score. Same
+      # reason the install job asserts that `verify` refuses a skill that was altered.
+      - name: Task leakage detector still detects
+        run: python3 tools/lint_task_leakage.py --self-test
+
+      # Static, stdlib, no container and no credential, so it belongs with the level 1
+      # gate rather than with the harbor jobs it reasons about. It fails a task whose
+      # instruction hands over the API symbols its own skill teaches: both arms of the
+      # differential then read the answer out of the prompt, so the task cannot
+      # discriminate — at the same price as one that can.
+      #
+      # 5 is the score of the worst task in the tree today. The budget is a ratchet
+      # rather than a target: it blocks a task worse than the worst one already here,
+      # grandfathers the twelve that leak, and comes down as those are rewritten — the
+      # self-test above fails if it does not. Because the score is an intersection, a
+      # pull request that only adds to a skill can turn one of them red; that is the
+      # check working, since a task stops discriminating when the skill grows into it.
+      - name: Task leakage
+        run: python3 tools/lint_task_leakage.py --fail-on-leak 5
+
       # Its own step because it is the one check that depends on someone else's
       # server. A 404 or 410 fails it — that is a pointer an agent would follow
       # into nothing — unless the link came from an imported body, where it warns

CONTRIBUTING.md

@@ -212,11 +212,13 @@ install:
 python3 tools/validate_skills.py                 # every offline check CI blocks on
 python3 tools/validate_skills.py --check-links   # also checks external URLs; needs network
 python3 tools/run_evals.py --validate            # only if you wrote evals/evals.json
+python3 tools/lint_task_leakage.py --fail-on-leak 5   # only if you wrote a Harbor task
 python3 tools/sync_external.py --check           # only if you imported a skill
 ```
 
-The first and third need no network. If they pass, the blocking checks left are about the
-repository rather than your text: the workflow linters and the installer round trip.
+Only `--check-links` and `sync_external.py --check` reach the network. If the offline ones
+pass, the blocking checks left are about the repository rather than your text: the workflow
+linters and the installer round trip.
 
 ## 5. If you are writing a new skill, add a Harbor task
 
@@ -245,6 +247,13 @@ task whose `instruction.md` contains the answer is passed with or without the sk
 measures nothing, at the same price as one that measures something. Keyless and offline,
 like the rest of the local gate.
 
+CI fails a task scoring above 5, which is what the worst task already here scores — so the
+budget stops a new task being worse than the worst one, and is not a number to aim at. Aim
+at zero: name the library, state the result you want, and leave the call to the agent. If
+you improve an existing instruction and the worst score in the tree drops, CI asks you to
+lower the budget in `.github/workflows/validate.yml` in the same pull request; the number is
+meant to ratchet down.
+
 If your skill cannot be exercised without an Intel GPU, say so in the pull request and a
 maintainer will decide — a task only that team's hardware can run is not a gate, it is a
 favour someone does.
@@ -271,6 +280,9 @@ Blocking, keyless, and runnable on a fork:
 - `skills.yaml` has an entry with a maintainer, and the catalog and the tree agree
 - the workflows themselves lint clean (`actionlint`, `zizmor`)
 - for a new skill: its Harbor task is solvable, oracle reward 1.0
+- no Harbor task's instruction gives away more than 5 points of its own skill's answer —
+  a point per API symbol the skill teaches, three per line of code copyable straight out
+  of the prompt
 - for an imported skill: `skills.yaml`, `.source.json` and `NOTICE` agree, and the copy is
   still byte-for-byte the pinned upstream commit
 - `npx … install` writes every skill in the catalog, and `verify` accepts each one and

MAINTAINERS.md

@@ -98,7 +98,10 @@ can reproduce is not evidence.
 are at the ceiling and cannot show a delta in either direction. Such tasks are marked
 `calibration: "ceiling"` in `suites.json` and no skill may cite an improvement from that
 run. `tools/lint_task_leakage.py` ranks how much of a task's answer its own instruction
-leaks, which is the usual cause.
+leaks, which is the usual cause, and `validate.yml` blocks above 5 — the score of the worst
+task in the tree, so the gate holds the line rather than clearing it. A task under the
+budget can still sit at a ceiling: leakage is necessary, not sufficient, and only the
+no-skill arm settles it.
 
 Before trusting a suite, run it against a deliberately falsified copy of the skill as one
 arm. A suite that scores a lying skill as highly as the real one is inert. Two cautions
@@ -220,7 +223,7 @@ the fix, the pin is the wrong pin.
 
 | Workflow | Job | Runs on | Blocks? |
 |---|---|---|---|
-| `validate.yml` | `validate` — `validate_skills.py`, `run_evals.py --validate`, link check | every PR | yes |
+| `validate.yml` | `validate` — `validate_skills.py`, `run_evals.py --validate`, task leakage and its self-test, link check | every PR | yes |
 | `validate.yml` | `install` — the installer resolves, lists, and installs from the catalog | every PR | yes |
 | `harbor-smoke.yml` | the oracle arm over every task in `tasks/` | PRs touching tasks or skills | yes |
 | `security.yml` | `actionlint`, `zizmor` | every PR | yes |
@@ -260,7 +263,7 @@ network and both say so when they cannot: `validate_skills.py --check-links` and
 | `run_evals.py` | validates eval files against their schema; scores recorded answers |
 | `compare_harbor_skill.py` | runs and reports the three-arm differential, with cost and time |
 | `check_harbor_job.py` | asserts a harbor run's trial count and reward floor |
-| `lint_task_leakage.py` | ranks how much of its own answer each task's instruction leaks |
+| `lint_task_leakage.py` | ranks how much of its own answer each task's instruction leaks; blocks above 5 in CI, and `--self-test` asserts against this tree that the detector behind that number still detects |
 | `behavior_digest.py` | digests the skill bytes a measurement was taken against, so a later edit to `SKILL.md` cannot leave `perf/` describing text that no longer exists |
 
 Two more exist for the imported skills: `sync_external.py` regenerates a copy from its pin

evaluation/harbor/README.md

@@ -319,11 +319,12 @@ no container, no model, no credentials.
 ```bash
 python3 tools/lint_task_leakage.py                 # every task, ranked
 python3 tools/lint_task_leakage.py --task dpnp-linalg-matmul --show
-python3 tools/lint_task_leakage.py --fail-on-leak 2
+python3 tools/lint_task_leakage.py --fail-on-leak 5   # what CI runs
+python3 tools/lint_task_leakage.py --self-test        # and what CI runs before it
 ```
 
-Over all fifteen tasks (2026-08-26): `dpnp-migration-replace-numpy` 5,
-`dpnp-device-fallback` 3, `dpnp-reduction-stats` 3, `mkl-fft-patch-numpy` 3,
+Over all fifteen tasks (2026-09-11): `dpnp-migration-replace-numpy` 5,
+`dpnp-device-fallback` 4, `dpnp-reduction-stats` 3, `mkl-fft-patch-numpy` 3,
 `dpnp-fft-pipeline` 2, `mkl-random-parallel-streams` 2, `dpnp-linalg-matmul` 1,
 `mkl-umath-coverage-report` 1, four `onetbb-*` 1, and three `onetbb-*` at 0. Every
 task that measured nothing in the scored run leaks; `dpnp-reduction-stats` hands over
@@ -339,9 +340,23 @@ nothing to give.
 
 Read a zero as "this task *can* discriminate", never as "it will" — a model that knows
 the answer from pre-training produces the same ceiling with a clean instruction, and
-only the `no_skill` arm can tell the two apart. The check is report-only by default,
-because every task in the repository leaks today; `--fail-on-leak N` is for a suite
-that has been cleaned up and wants to stay that way.
+only the `no_skill` arm can tell the two apart. The check reports by default and
+`--fail-on-leak N` makes it block; `validate.yml` runs it at 5, the score of the worst
+task above. That grandfathers the twelve tasks that leak today and blocks a new one from
+being worse than any of them, so the number is a ratchet to bring down as those twelve
+are rewritten — not a standard a new task should aim at. Note that the score is an
+intersection, so growing a skill can push a task that was under the budget over it, with
+no change to the task at all: the task really has stopped discriminating, and the fix is
+the instruction rather than the budget.
+
+The ratchet is not left to good intentions. `--self-test` asserts, against the skills and
+tasks in this tree, that the detector still detects — a dotted call, a `::` call and a
+keyword argument are all still seen, an alias bound twice keeps both meanings, one more
+given-away symbol costs exactly one point and a pasted line costs three — and that the
+budget in `validate.yml` is still the worst task's score exactly. So a regex that stopped
+matching fails loudly instead of reporting a clean zero for every task, and improving the
+worst instruction turns CI red until the budget comes down with it. It writes nothing; the
+two injections are made to a copy of the instruction in memory.
 
 Keep the task revision, agent, model, attempt count, and timeouts identical across
 arms — the only difference may be the skill. All three arms in one command:

tools/lint_task_leakage.py

@@ -60,6 +60,10 @@
 REPO_ROOT = Path(__file__).resolve().parent.parent
 TASKS_DIR = REPO_ROOT / "evaluation" / "harbor" / "tasks"
 SKILLS_DIR = REPO_ROOT / "skills"
+# --self-test reads the gate's budget out of the workflow rather than carrying a second
+# copy of the number, so the two cannot drift apart.
+WORKFLOW = REPO_ROOT / ".github" / "workflows" / "validate.yml"
+GATE_BUDGET = re.compile(r"--fail-on-leak\s+(\d+)")
 
 # A task must be able to name the library it is about, so a bare module or a common
 # alias is premise rather than answer. Anything dotted onto them is the answer.
@@ -218,6 +222,121 @@ def audit(task_dir: Path, allow: set[str]) -> dict | None:
     }
 
 
+def _score(taught: set[str], taught_lines: set[str], instruction: str) -> int:
+    """audit()'s score, for an instruction held in memory rather than read from disk."""
+    return len(taught & symbols(instruction)) + 3 * len(taught_lines & code_lines(instruction))
+
+
+def self_test() -> int:
+    """Assert the detector still detects, against the skills and tasks in this tree.
+
+    Every way this check can break is silent. A regex that stops matching, a PREMISE
+    entry that swallows a real symbol, an alias map that resolves the wrong way: each of
+    them prints a clean zero for every task and a green gate, which reads exactly like a
+    repository whose tasks are all sound. A leakage budget is worth what the detector
+    behind it is worth, so the properties it rests on are asserted rather than assumed —
+    the same reason validate.yml asserts that `verify` refuses an installed skill that
+    was altered.
+
+    Real content, because a fixture proves the fixture: the assertions below run against
+    `skills/` and `evaluation/harbor/tasks/` as committed. Nothing is written — the two
+    injections are made to a copy of the instruction text in memory.
+    """
+    failures: list[str] = []
+
+    def check(name: str, ok: bool, detail: str) -> None:
+        print(f"{'ok  ' if ok else 'FAIL'} {name} - {detail}")
+        if not ok:
+            failures.append(name)
+
+    python_skill, cxx_skill = skill_text("dpnp-quickstart"), skill_text("onetbb-quickstart")
+    if python_skill is None or cxx_skill is None:
+        sys.exit("FAIL --self-test needs skills/dpnp-quickstart and skills/onetbb-quickstart")
+
+    taught = symbols(python_skill)
+    check("a Python skill teaches a dotted symbol", "dpnp.std" in taught,
+          f"dpnp-quickstart teaches {len(taught)} symbols")
+
+    qualified = {symbol for symbol in symbols(cxx_skill) if "::" in symbol}
+    check("a C++ skill teaches a :: symbol", "tbb::parallel_for" in qualified,
+          f"onetbb-quickstart teaches {len(qualified)} of them")
+
+    check("a keyword argument is taught, not only a call", "device=" in taught,
+          "dpnp-quickstart teaches `device=`, which is what dpnp-device-fallback gives away")
+
+    check("an alias bound twice keeps both meanings", aliases(python_skill).get("np") == {"dpnp", "numpy"},
+          "dpnp-quickstart writes `import dpnp as np` and `import numpy as np` in one document")
+
+    reports = [
+        report
+        for report in (audit(path, set()) for path in sorted(TASKS_DIR.iterdir()) if path.is_dir())
+        if report is not None and "error" not in report
+    ]
+    scored = [report for report in reports if report["score"] > 0]
+    worst_score = max(report["score"] for report in reports)
+
+    # The budget is only a ratchet if it tracks the tree. Read it out of the workflow and
+    # require it to be the worst task's score exactly: too high and the gate has slack
+    # nobody voted for, too low and CI is red on content that was already merged. Cleaning
+    # up the worst task therefore turns this red, which is the check asking for the number
+    # to come down with it.
+    budgets = (
+        {int(found) for found in GATE_BUDGET.findall(WORKFLOW.read_text(encoding="utf-8"))}
+        if WORKFLOW.is_file()
+        else set()
+    )
+    check("the CI budget is the worst task in the tree", budgets == {worst_score},
+          f"validate.yml runs --fail-on-leak {sorted(budgets) or 'nothing'}, worst of "
+          f"{len(scored)} leaking tasks scores {worst_score}"
+          + ("" if budgets == {worst_score} else " - move the budget in validate.yml to match"))
+
+    given_away = {symbol for report in reports for symbol in report["leaked_symbols"]}
+    unqualified = sorted(s for s in given_away if not SEPARATOR.search(s) and not s.endswith("="))
+    check("no bare module is reported as an answer", not unqualified,
+          ", ".join(unqualified) or f"all {len(given_away)} are qualified calls or keyword arguments")
+
+    # The bite, on the task that is already worst: the budget is only a budget if one
+    # more given-away symbol crosses it.
+    worst = max(reports, key=lambda report: report["score"])
+    instruction = (TASKS_DIR / worst["task"] / "instruction.md").read_text(encoding="utf-8")
+    skill = skill_text(worst["skill"]) or ""
+    taught, taught_lines = symbols(skill), code_lines(skill)
+    base = _score(taught, taught_lines, instruction)
+    check(f"{worst['task']} scores the same in memory", base == worst["score"],
+          f"on disk {worst['score']}, in memory {base}")
+
+    untold = sorted(taught - symbols(instruction))
+    unshared = sorted(taught_lines - code_lines(instruction))
+    if not untold or not unshared:
+        sys.exit(f"FAIL --self-test needs a symbol and a line {worst['task']} does not already give away")
+
+    # A library call rather than whatever sorts first: the regex also matches a filename
+    # like `SKILL.md`, and injecting one of those would assert the arithmetic while
+    # proving nothing about the thing being detected.
+    call = next((s for s in untold if SEPARATOR.split(s)[0] in PREMISE), untold[0])
+    with_symbol = _score(taught, taught_lines, f"{instruction}\n\nUse `{call}` for this.\n")
+    check("one more given-away symbol costs one point", with_symbol == base + 1,
+          f"`{call}` added to {worst['task']}: {base} -> {with_symbol}, so a budget of {base} fails")
+
+    pasted = f"{instruction}\n\n```python\n{unshared[0]}\n```\n"
+    shared_before, shared_after = (
+        len(taught_lines & code_lines(instruction)),
+        len(taught_lines & code_lines(pasted)),
+    )
+    with_line = _score(taught, taught_lines, pasted)
+    check("a copyable line is caught and costs at least three",
+          shared_after == shared_before + 1 and with_line >= base + 3,
+          f"one line of the skill pasted into {worst['task']}: score {base} -> {with_line}, "
+          f"shared lines {shared_before} -> {shared_after}")
+
+    if failures:
+        print(f"\nFAIL {len(failures)} self-test(s): " + ", ".join(failures), file=sys.stderr)
+        return 1
+    print("\nOK the detector detects. A regex that stopped matching would fail here rather")
+    print("than report a clean zero for every task and leave the budget passing anything.")
+    return 0
+
+
 def main() -> int:
     parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
     parser.add_argument("--task", action="append", default=[], help="limit to these task names")
@@ -234,15 +353,26 @@ def main() -> int:
         type=int,
         default=None,
         metavar="N",
-        help="exit nonzero for any task scoring above N. Omit to report only, which is "
-        "the default because the fifteen tasks in this repository all leak today.",
+        help="exit nonzero for any task scoring above N. Omit to report only. CI runs "
+        "with 5, the score of the worst task here today, so the gate is a ratchet: it "
+        "stops a task arriving worse than the worst one already in the tree.",
     )
     parser.add_argument("--json", action="store_true", help="machine-readable output")
+    parser.add_argument(
+        "--self-test",
+        action="store_true",
+        help="assert the detector still detects, against the skills and tasks in this "
+        "tree. Writes nothing. Run it before the gate: a broken detector reports a clean "
+        "zero for every task, which is indistinguishable from a clean repository.",
+    )
     args = parser.parse_args()
 
     if not TASKS_DIR.is_dir():
         sys.exit(f"FAIL no {TASKS_DIR.relative_to(REPO_ROOT).as_posix()}")
 
+    if args.self_test:
+        return self_test()
+
     wanted = set(args.task)
     reports = []
     for task_dir in sorted(TASKS_DIR.iterdir()):