ci: block a Harbor task that gives away its own answer

[xaleryb]

tools/lint_task_leakage.py has been in the tree since the catalog landed and is wired into no workflow. So a Harbor task whose instruction.md hands over the API symbols its own skill teaches can merge: both arms of the differential read the answer out of the prompt, both score 1.0, and the task measures nothing at the price of one that measures something. That is how the first scored run of dpnp-quickstart put all five of its tasks at a ceiling — visible statically, without a container or a credential, but only if something runs the check.

What this does

validate.yml gains two steps in the validate job, both stdlib-only and fork-safe:

- name: Task leakage detector still detects
  run: python3 tools/lint_task_leakage.py --self-test

- name: Task leakage
  run: python3 tools/lint_task_leakage.py --fail-on-leak 5

Why the budget is 5

Measured over all fifteen tasks as committed:

task	score	symbols	shared lines
dpnp-migration-replace-numpy	5	dpnp.arange, dpnp.asnumpy, np.arange, numpy.arange, numpy.dot	0
dpnp-device-fallback	4	device=, dpnp.arange, dpnp.sum, y=	0
dpnp-reduction-stats	3	dpnp.mean, dpnp.std, dpnp.sum	0
mkl-fft-patch-numpy	3	mkl_fft.is_patched, np.fft.fft.__module__, numpy.fft	0
dpnp-fft-pipeline, mkl-random-parallel-streams	2		0
seven others	1		0
onetbb-flow-graph, onetbb-parallel-scan, onetbb-parallel-sort	0		0

5 is the tightest budget that is green on this tree, and no task shares a verbatim line of code with its skill. The budget is a ratchet rather than a target: it grandfathers the twelve tasks that leak today and blocks a new one from arriving worse than any of them. Bringing it below 5 means rewriting instructions, which is a separate change.

Because the score is an intersection, a pull request that only adds to a skill can push a task over the budget with no edit to the task at all. That is the check working — the task really has stopped discriminating, and the fix is the instruction, not the number.

Why there is a self-test

A gate is worth what the detector behind it is worth, and every way this detector can break is silent: a regex that stops matching reports a clean zero for every task, which reads exactly like a repository whose tasks are all sound. So --self-test asserts the detector against the skills and tasks in this tree — the same reason the install job asserts that verify refuses a skill that was altered. It writes nothing; both injections are made to a copy of the instruction text in memory.

ok  a Python skill teaches a dotted symbol - dpnp-quickstart teaches 61 symbols
ok  a C++ skill teaches a :: symbol - onetbb-quickstart teaches 6 of them
ok  a keyword argument is taught, not only a call - `device=`, which dpnp-device-fallback gives away
ok  an alias bound twice keeps both meanings - `import dpnp as np` and `import numpy as np` in one document
ok  the CI budget is the worst task in the tree - validate.yml runs --fail-on-leak [5], worst of 12 leaking tasks scores 5
ok  no bare module is reported as an answer - all 18 are qualified calls or keyword arguments
ok  dpnp-migration-replace-numpy scores the same in memory - on disk 5, in memory 5
ok  one more given-away symbol costs one point - `dpctl.get_devices` added: 5 -> 6, so a budget of 5 fails
ok  a copyable line is caught and costs at least three - score 5 -> 12, shared lines 0 -> 1

The budget assertion reads --fail-on-leak out of validate.yml rather than carrying a second copy of the number, and requires it to equal the worst task's score exactly. So the gate cannot keep slack nobody voted for, and improving the worst instruction turns CI red until the budget comes down with it — the ratchet is enforced rather than promised.

Six ways of breaking the detector were tried, and each one fails the self-test:

mutation	caught by
symbol regex matches nothing	dotted symbol, :: symbol, budget matches tree, +1 point
keyword regex matches nothing	a keyword argument is taught
PREMISE swallows a real call (dpnp.std)	dotted symbol
alias map keeps only the last binding	alias bound twice, budget matches tree
code_lines() returns nothing	the guard: no unshared line left to inject
budget drifts above the tree (reads 10)	the CI budget is the worst task in the tree

Two recorded numbers were wrong

Corrected here, since wiring the check up is what stops a recorded number drifting from the tool again:

• evaluation/harbor/README.md put dpnp-device-fallback at 3 where the committed tool prints 4. Nothing changed under it — the task, the skill and the tool all arrived in the same commit — so this was a transcription error from the start.
• the tool's own --fail-on-leak help said the fifteen tasks "all leak today", where three score zero. Its example also showed --fail-on-leak 2, which fails on this tree.

What is not claimed

A task under the budget can still sit at a ceiling. Leakage is necessary, not sufficient — a model that knows the answer from pre-training produces the same tie with a spotless instruction, and only the no_skill arm tells the two apart. This check says "this task cannot discriminate", never "this task will".

Docs in the same change

• CONTRIBUTING.md — the command in the local gate list, the rule in "What CI checks", and in the task section: aim at zero, not at 5, and lower the budget in the same PR if you improve the worst instruction
• MAINTAINERS.md — the CI table row, the tools row, and the caveat above next to the tie-is-a-verdict-on-the-suite rule
• evaluation/harbor/README.md — the corrected ranking, --self-test in the command block, and what the budget does and does not mean

Checks run locally

validate_skills.py, run_evals.py --validate, --self-test, --fail-on-leak 5, --json, --help all exit 0. validate_skills.py --check-links exits 0 with 77 pre-existing warnings (404s in imported bodies, timeouts through a proxy). The workflow YAML parses and the step order is Skill structure and catalog → Eval schema → Task leakage detector still detects → Task leakage → Link check → Imported skills match their pinned upstream. actionlint and zizmor are not installed locally; both new steps are plain run: lines with no expressions, and security.yml will confirm.

---

Opened from a fork, replacing #21, which had the same commit (711630b) on a branch pushed into this repository by mistake. No review had happened there.

[napetrov]

bad way to number things in constantly changing documentation