Skip to content

Commit f81a23f

Browse files
donislawdevclaude
andcommitted
An npm package that gives Playwright tests a generated set, and an example of what it finds
npm/testing-files-generator plans a set with tfg while a spec file loads, writing nothing, so every file becomes a test with a title, tags and the sentence that says what the file is for. The first test that asks for a path writes the set, once for every worker: each writes a copy and the first to move it to its name wins, the others take it and remove theirs through tfg cleanup. Later runs take the same files, keyed by the arguments as well as the recipe hash, which leaves out the seed and --clean. check compares what the system did with the manifest and throws with that sentence. Unspecified files are skipped, never passed. No dependencies, no install script, MPL-2.0. Not published yet. examples/playwright is an upload form with four mistakes. The generated set finds all four through the page and the API, the tests written with Buffer.alloc find none, and with the mistakes fixed nothing fails. CI runs the package's tests on three systems and the example on Linux. Guards hold the package to the platforms the release builds, the closed lists and fields of the manifest, and the punctuation of the text it shows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 4fbb035 commit f81a23f

31 files changed

Lines changed: 3079 additions & 0 deletions

‎.github/workflows/ci.yml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1482,6 +1482,66 @@ jobs:
14821482
run: go test -tags "$(cat .github/build-tags)" ./internal/guard/ -count=1 -run 'ReferenceTool' -v
14831483
shell: bash
14841484

1485+
npm:
1486+
name: npm package on ${{ matrix.os }}
1487+
# The npm package (npm/testing-files-generator) runs tfg and reads what it
1488+
# writes from JavaScript, so nothing in the Go suite runs it. Its own tests
1489+
# run here against the binary this commit builds, on the three systems a
1490+
# release has binaries for - Windows above all, because the rename that
1491+
# settles a race between test workers is refused there in ways it is not
1492+
# elsewhere. Node 22 is the oldest the package declares and Playwright
1493+
# supports, so it is the one tested.
1494+
#
1495+
# The Playwright example runs on Linux only. It is the package's acceptance
1496+
# test: an upload form with four mistakes, where the generated set has to
1497+
# find all four, the tests written with Buffer.alloc have to find none, and
1498+
# with the mistakes fixed nothing may fail - that last run is the control
1499+
# without which a red test could come from the tests. One browser on one
1500+
# system is enough to show that, and the package itself is tested on all
1501+
# three. docs/PLAYWRIGHT-2026-11-04.md.
1502+
timeout-minutes: 20
1503+
strategy:
1504+
fail-fast: false
1505+
matrix:
1506+
os: [ubuntu-latest, windows-latest, macos-latest]
1507+
runs-on: ${{ matrix.os }}
1508+
env:
1509+
CGO_ENABLED: "0"
1510+
steps:
1511+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
1512+
with:
1513+
persist-credentials: false
1514+
1515+
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
1516+
with:
1517+
go-version: ${{ env.GO_VERSION }}
1518+
check-latest: false
1519+
cache: true
1520+
1521+
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
1522+
with:
1523+
node-version: "22"
1524+
1525+
- name: build tfg where the package's tests and the example look for it
1526+
run: go build -tags "$(cat .github/build-tags)" -o "tfg$(go env GOEXE)" ./cmd/tfg
1527+
shell: bash
1528+
1529+
- name: the package's own tests
1530+
working-directory: npm/testing-files-generator
1531+
run: node --test "test/*.test.js"
1532+
shell: bash
1533+
1534+
- name: the Playwright example finds the four mistakes, and nothing once they are fixed
1535+
if: runner.os == 'Linux'
1536+
working-directory: examples/playwright
1537+
run: |
1538+
set -euo pipefail
1539+
npm ci --no-audit --no-fund
1540+
npx playwright install --with-deps chromium
1541+
npx tsc --noEmit
1542+
node check.mjs
1543+
shell: bash
1544+
14851545
fuzz:
14861546
name: fuzzing with a time budget
14871547
# Weekly and by hand, never on a push. Fuzzing searches without end, so a

‎examples/playwright/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
node_modules/
2+
results/
3+
test-results/

‎examples/playwright/README.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# An upload form, the tests it usually gets, and the tests a generated set gives it
2+
3+
`app/server.mjs` is an upload form with four mistakes, each one a mistake real upload forms have:
4+
5+
1. The server never checks the size. Only the page does.
6+
2. The content and the extension are each checked against the allowed types, and never against each other, so a PDF named `.jpg` goes through.
7+
3. An allowed extension anywhere in the name counts, so `invoice.jpg.exe` is taken for a picture.
8+
4. The page refuses a file of exactly the limit, which is the largest size allowed.
9+
10+
`tests/typical.spec.ts` is what such a form usually gets: a real picture that should go through, and files made in the test with `Buffer.alloc` that should not. Every one of those tests passes. The file over the limit is zeros rather than a PDF, so the server turns it away for what it holds, and the test meant to prove the size limit passes on a server that has none.
11+
12+
`tests/upload.spec.ts` sends the `upload-validation` set from [testing-files-generator](../../npm/testing-files-generator) through the page and straight to the server. It finds all four mistakes, and each failure says what it means.
13+
14+
## Run it
15+
16+
From the root of the repository, build tfg first. The example finds it there.
17+
18+
```bash
19+
go build -tags "$(cat .github/build-tags)" -o tfg.exe ./cmd/tfg
20+
```
21+
22+
```bash
23+
cd examples/playwright && npm install && npx playwright install chromium
24+
```
25+
26+
```bash
27+
npx playwright test
28+
```
29+
30+
Eleven tests fail, five through the page and six through the API, and every failure names its file and the mistake it shows:
31+
32+
```
33+
1mb_over_1b.jpg was taken, and it should have been turned away (size_limit).
34+
A JPEG file 1 B over the limit of 1mb. Your system should turn it away - taking it means
35+
the limit is not enforced, or enforced only in the browser.
36+
```
37+
38+
Ten are skipped, because what a form does with an empty file or a name full of emoji is its own policy, and the set says so rather than pass them.
39+
40+
`FIXED=1 npx playwright test` corrects all four mistakes, and nothing fails. `npm run check` runs both and says whether the example still shows what it is for. The repository's CI runs that.

‎examples/playwright/app/index.html‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
<!doctype html>
2+
<html lang="en">
3+
<head><meta charset="utf-8"><title>Upload</title></head>
4+
<body data-limit="{{limit}}" data-fixed="{{fixed}}">
5+
<form id="upload">
6+
<label for="file">File</label>
7+
<input type="file" id="file" name="file">
8+
<button type="submit">Upload</button>
9+
</form>
10+
<p role="status" id="status"></p>
11+
<script src="/page.js"></script>
12+
</body>
13+
</html>

‎examples/playwright/app/page.js‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
// The form's own check, in the browser, before anything is sent.
2+
3+
const limit = Number(document.body.dataset.limit);
4+
const fixed = document.body.dataset.fixed === '1';
5+
6+
document.getElementById('upload').addEventListener('submit', async (event) => {
7+
event.preventDefault();
8+
const status = document.getElementById('status');
9+
const file = document.getElementById('file').files[0];
10+
// Mistake 4: a file of exactly the limit is refused. The limit is the
11+
// largest size allowed, so the comparison should be >.
12+
const tooLarge = fixed ? file.size > limit : file.size >= limit;
13+
if (tooLarge) {
14+
status.textContent = 'Refused (size_limit)';
15+
return;
16+
}
17+
const body = new FormData();
18+
body.append('file', file);
19+
const response = await fetch('/upload', { method: 'POST', body });
20+
const answer = await response.json();
21+
status.textContent = response.ok ? 'Uploaded' : `Refused (${answer.reason})`;
22+
});

‎examples/playwright/app/server.mjs‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
// An upload form with four mistakes made on purpose, each one a mistake real
2+
// upload forms have. playwright.config.ts starts it. FIXED=1 corrects all four,
3+
// which is how check.mjs shows that the red tests come from the mistakes and
4+
// not from the tests.
5+
//
6+
// It uses nothing outside Node, so the example has no server dependency.
7+
8+
import { readFileSync } from 'node:fs';
9+
import http from 'node:http';
10+
import { Readable } from 'node:stream';
11+
12+
const limit = Number(process.env.UPLOAD_LIMIT_BYTES);
13+
const port = Number(process.env.PORT);
14+
const fixed = process.env.FIXED === '1';
15+
const allowed = ['jpg', 'png', 'pdf'];
16+
17+
if (!Number.isInteger(limit) || !Number.isInteger(port)) {
18+
throw new Error('Set UPLOAD_LIMIT_BYTES and PORT. playwright.config.ts sets both.');
19+
}
20+
21+
// What a file holds, read from its first bytes rather than from its name.
22+
function sniff(bytes) {
23+
if (bytes[0] === 0xff && bytes[1] === 0xd8 && bytes[2] === 0xff) {
24+
return 'jpg';
25+
}
26+
if (bytes.subarray(0, 8).equals(Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]))) {
27+
return 'png';
28+
}
29+
if (bytes.subarray(0, 5).toString('latin1') === '%PDF-') {
30+
return 'pdf';
31+
}
32+
return '';
33+
}
34+
35+
function extensionOf(name) {
36+
const lower = name.toLowerCase();
37+
if (fixed) {
38+
const dot = lower.lastIndexOf('.');
39+
return dot > 0 ? lower.slice(dot + 1) : '';
40+
}
41+
// Mistake 3: an allowed extension anywhere in the name counts, so
42+
// invoice.jpg.exe is taken for a picture.
43+
return allowed.find((ext) => lower.includes(`.${ext}`)) ?? '';
44+
}
45+
46+
function verdict(name, bytes) {
47+
// Mistake 1: the server never looks at the size. The page does, so a test
48+
// through the page sees a limit that a request straight to the server does not.
49+
if (fixed && bytes.length > limit) {
50+
return { status: 413, reason: 'size_limit' };
51+
}
52+
const extension = extensionOf(name);
53+
if (!allowed.includes(extension)) {
54+
return { status: 422, reason: 'extension_rule' };
55+
}
56+
const content = sniff(bytes);
57+
if (!allowed.includes(content)) {
58+
return { status: 415, reason: 'content_malformed' };
59+
}
60+
// Mistake 2: the content and the extension are each checked against the
61+
// allowed list and never against each other, so a PDF named .jpg goes through.
62+
if (fixed && content !== extension) {
63+
return { status: 415, reason: 'mime_mismatch' };
64+
}
65+
return { status: 201 };
66+
}
67+
68+
// The page carries the limit and the mode, and page.js reads them from it.
69+
const page = readFileSync(new URL('index.html', import.meta.url), 'utf8')
70+
.replace('{{limit}}', String(limit))
71+
.replace('{{fixed}}', fixed ? '1' : '');
72+
const script = readFileSync(new URL('page.js', import.meta.url), 'utf8');
73+
74+
async function upload(req, res) {
75+
const request = new Request(`http://localhost${req.url}`, {
76+
method: 'POST', headers: req.headers, body: Readable.toWeb(req), duplex: 'half',
77+
});
78+
const file = (await request.formData()).get('file');
79+
if (!file || typeof file === 'string') {
80+
return answer(res, 400, { reason: 'none' });
81+
}
82+
const { status, reason } = verdict(file.name, Buffer.from(await file.arrayBuffer()));
83+
return answer(res, status, reason ? { reason } : { stored: file.name });
84+
}
85+
86+
function answer(res, status, body) {
87+
res.writeHead(status, { 'content-type': 'application/json' });
88+
res.end(JSON.stringify(body));
89+
}
90+
91+
http.createServer((req, res) => {
92+
if (req.method === 'GET' && req.url === '/') {
93+
res.writeHead(200, { 'content-type': 'text/html; charset=utf-8' });
94+
res.end(page);
95+
return;
96+
}
97+
if (req.method === 'GET' && req.url === '/page.js') {
98+
res.writeHead(200, { 'content-type': 'text/javascript; charset=utf-8' });
99+
res.end(script);
100+
return;
101+
}
102+
if (req.method === 'POST' && req.url === '/upload') {
103+
upload(req, res).catch((err) => answer(res, 500, { reason: 'none', error: String(err) }));
104+
return;
105+
}
106+
res.writeHead(404);
107+
res.end();
108+
}).listen(port, () => console.log(`upload form on http://localhost:${port}${fixed ? ', mistakes fixed' : ''}`));

‎examples/playwright/check.mjs‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
// Runs the example twice and says whether it shows what it is for.
2+
//
3+
// With the four mistakes, the tests from the generated set fail on exactly the
4+
// files each mistake lets through or turns away, and every typical test passes.
5+
// With FIXED=1, nothing fails. The second run is the control: without it, red
6+
// tests could come from the tests themselves rather than from the app.
7+
8+
import { spawnSync } from 'node:child_process';
9+
import { readFileSync } from 'node:fs';
10+
import { createRequire } from 'node:module';
11+
12+
// Playwright's own command line, run by this Node rather than through npx and
13+
// a shell, which Windows would need for npx.
14+
const playwright = createRequire(import.meta.url).resolve('@playwright/test/cli');
15+
16+
// Each mistake in app/server.mjs, and the tests it turns red.
17+
const mistakes = {
18+
'through the page': {
19+
'1mb_at_limit.jpg': 4,
20+
'pdf_as_jpg.jpg': 2,
21+
'png_as_pdf.pdf': 2,
22+
'jpg_as_png.png': 2,
23+
'invoice.jpg.exe': 3,
24+
},
25+
'through the API': {
26+
'1mb_over_1b.jpg': 1,
27+
'1mb_far_over_2x.jpg': 1,
28+
'pdf_as_jpg.jpg': 2,
29+
'png_as_pdf.pdf': 2,
30+
'jpg_as_png.png': 2,
31+
'invoice.jpg.exe': 3,
32+
},
33+
};
34+
35+
function run(fixed) {
36+
spawnSync(process.execPath, [playwright, 'test', '--reporter=json'], {
37+
stdio: ['ignore', 'ignore', 'inherit'],
38+
env: { ...process.env, FIXED: fixed ? '1' : '', PLAYWRIGHT_JSON_OUTPUT_NAME: 'results/check.json' },
39+
});
40+
const report = JSON.parse(readFileSync('results/check.json', 'utf8'));
41+
const tests = [];
42+
const walk = (suite, titles) => {
43+
for (const spec of suite.specs ?? []) {
44+
for (const one of spec.tests) {
45+
tests.push({ file: spec.file, describe: titles.at(-1) ?? '', title: spec.title, status: one.status });
46+
}
47+
}
48+
for (const child of suite.suites ?? []) {
49+
walk(child, [...titles, child.title]);
50+
}
51+
};
52+
for (const suite of report.suites) {
53+
walk(suite, []);
54+
}
55+
if (report.errors?.length) {
56+
throw new Error(`The run did not get as far as its tests: ${report.errors.map((e) => e.message).join('\n')}`);
57+
}
58+
return tests;
59+
}
60+
61+
const problems = [];
62+
const say = (problem) => problems.push(problem);
63+
64+
for (const fixed of [false, true]) {
65+
const mode = fixed ? 'with the mistakes fixed' : 'with the four mistakes';
66+
const tests = run(fixed);
67+
const ran = tests.filter((t) => t.status !== 'skipped');
68+
const generated = tests.filter((t) => t.file.endsWith('upload.spec.ts'));
69+
if (ran.length === 0 || generated.every((t) => t.status === 'skipped')) {
70+
say(`${mode}: no test from the generated set ran`);
71+
}
72+
for (const t of tests) {
73+
const name = t.title.split(' - expected ')[0];
74+
const mistake = mistakes[t.describe]?.[name];
75+
const shouldFail = !fixed && mistake !== undefined && t.file.endsWith('upload.spec.ts');
76+
const failed = t.status === 'unexpected';
77+
if (shouldFail && !failed) {
78+
say(`${mode}: ${t.describe} - ${name} should be red for mistake ${mistake} and is ${t.status}`);
79+
}
80+
if (!shouldFail && failed) {
81+
say(`${mode}: ${t.describe ? `${t.describe} - ` : ''}${t.title} failed, and nothing in the app should make it`);
82+
}
83+
}
84+
const red = tests.filter((t) => t.status === 'unexpected').length;
85+
const skipped = tests.filter((t) => t.status === 'skipped').length;
86+
console.log(`${mode}: ${red} red, ${tests.length - red - skipped} green, ${skipped} skipped as unspecified`);
87+
}
88+
89+
const found = new Set(Object.values(mistakes).flatMap((files) => Object.values(files)));
90+
for (const n of [1, 2, 3, 4]) {
91+
if (!found.has(n)) {
92+
say(`mistake ${n} has no test that finds it`);
93+
}
94+
}
95+
96+
if (problems.length > 0) {
97+
console.error(`The example does not show what it is for:\n- ${problems.join('\n- ')}`);
98+
process.exit(1);
99+
}
100+
console.log('The generated set finds all four mistakes, the typical tests find none, and with the mistakes fixed nothing is red.');

0 commit comments

Comments
 (0)