perf: one filler instead of thirteen, and eight bytes per draw instead of one (#56)

* perf: one listing instead of two questions a file, and one collection instead of two

Three findings from the performance report, and two of them came back with the
report's own suggestion measured and refused.

preflight asked the filesystem twice for every planned file. It reads the
output directory once. A run of 100 000 files with --dry-run goes from
15.7-19.7 s to 0.49-0.56 s, order alternated - the check was 97% of it. Ten
thousand names measured on their own: 1.937 s of stat calls against 8.9 ms for
one listing.

That is also a fix rather than only a speedup, and the guard for it says so.
os.Stat follows a link, so a link pointing at nothing answered "no name here"
and the run replaced it without a word. A directory ENTRY is what a taken name
is, whatever it points at. With the old question put back, the new guard
reports that the run went ahead over a name somebody else's link was holding.

A directory that cannot be LISTED is still asked about file by file. Both
systems allow write permission without read, a run into such a directory has
always worked, and reading nothing there and calling it empty would let the run
write over what is inside. That fallback has its own guard, which skips on
Windows because denying a listing there needs an ACL.

The plan ceiling forced a collection to take every reading, so a run of one
kilobyte paid for two of them - measured with GODEBUG=gctrace=1, exactly two on
every run however small. It asks /gc/heap/allocs:bytes first, at 251 ns against
519 us, and only collects when that says it might be over. The shortcut is
sound by an inequality rather than by an estimate: the live heap cannot have
grown by more than has been allocated.

The report asked for /gc/heap/live:bytes and that metric is WRONG here. It
reports the heap as of the last collection, and measured on 2026-09-05 all four
existing ceiling guards stay green with it, because 25 MB of allocation makes
the collector run on its own and the lagging reading catches up by luck. With
the collector switched off the luck goes: a plan six times the ceiling is
accepted. The new guard turns the collector off for exactly that reason.

hashFile reads in 256 KiB pieces. The report asked for a 1 MB buffer through
io.CopyBuffer and that does nothing at all: os.File implements io.WriterTo, so
CopyBuffer hands it the whole job and throws the buffer away - 128 KiB, 256 KiB
and 1 MiB with a plain file all take the same 167-172 ms that io.Copy takes.
Hidden behind a reader that offers only Read, 256 KiB takes 161 ms against 199.
The size is measured too: 64 KiB is 182 ms and nothing above a quarter of a
megabyte can be told apart, so sixteen workers cost four megabytes.

planChildren is sized up front, since the total is known from the groups.

TotalBytes being walked twice is NOT done, and that is a measurement rather
than an oversight: one walk over 100 000 planned files has a median of 0 s and
a maximum of 541 us, so two of them cost half a millisecond of a nineteen
second run. Widening a signature for that would be a change nothing can see.

engine.go went past the length ceiling, and the guard asks for a split by what
the parts do rather than for a bigger number - so preflight and the questions
it asks about names are their own file now. Two ceilings came down with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* perf: one filler instead of thirteen, and eight bytes per draw instead of one

Thirteen packages each carried their own copy of the bulk random filler
loop, in four different shapes. Measured over 64 MiB through a 32 KiB
buffer, interleaved with the order reversed between repetitions:

  one draw per byte              182 MB/s   zip, targz, wav
  eight via a temporary array   1468 MB/s   bmp, gif, ico, opc, png, tiff
  eight via a shift loop         845 MB/s   avif, jpg, jxl, webp
  one store into the buffer     2499 MB/s   the shape they all use now

All thirteen now call core.FillRandomBE or core.FillRandomLE. Two
functions rather than one with a flag, because choosing the wrong byte
order is not a style mistake - it silently rewrites every file a format
has ever produced, and an argument would put that one typo away.

BREAKING: zip, targz and wav files have different bytes. Their padding
is where those formats spend almost the whole file, so almost every byte
changes. Size, structure and readability are untouched. End to end on
64 MB, ranges disjoint: zip 2.81x, targz 2.74-3.32x.

wav is in that list for uniformity and not for speed, by the owner's
decision after the measurement: its padding is audio modulo the frame
size, so exactly two bytes of a WAV differ and the time is unchanged.

The other ten packages moved with no byte change at all - the shift loop
IS little endian, which the performance report did not notice, so those
four collapse to one store for free. Checked across 24 formats at five
sizes and two seeds: three moved, twenty identical.

Also closes a blind spot the golden set had. Forcing the filler to emit a
constant moved 33 of the 54 pinned cases and not one WAV: wav_32kib lands
on a size the audio fills exactly and never reaches the filler, so that
path had no pinned witness. wav_with_the_padding_chunk is that witness.

Guard: TestBulkRandomBytesComeFromOnePlace, two mutations, both caught.
It names the one honest UintN caller as an exception and fails if that
exception outlives its code. A third mutation proves the golden set
covers the shared filler's short tail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Author: donislawdev

CHANGELOG.md

@@ -14,6 +14,29 @@ because it turns other people's test suites red.
 
 ## [Unreleased]
 
+### Breaking
+
+- **ZIP, TAR.GZ and WAV files have different bytes.** The padding these three
+  formats write is now drawn from the random generator eight bytes at a time
+  instead of one byte at a time. For a ZIP or a TAR.GZ that padding is almost
+  the whole file, so almost every byte changes.
+
+  Nothing a reader can see is different. The size is the same to the byte, the
+  structure is the same, the same tools open the same files. What changes is the
+  content of the padding, so a hash you recorded from an earlier version will
+  not match one you generate now.
+
+  This is what makes large archives faster to produce. Measured on 64 MB files,
+  runs interleaved: ZIP 2.8 times faster, TAR.GZ 2.7 to 3.3 times faster.
+
+  WAV is on this list for consistency rather than for speed. Its padding is at
+  most a few bytes per file, so only those bytes differ and the time to produce
+  one is unchanged.
+
+  Every other format keeps exactly the bytes it had. Twelve of them share the
+  same new code and were checked against their recorded hashes and across every
+  format at five sizes and two seeds.
+
 ### Changed
 
 - **`verify` and `cleanup` read the files over several threads, so checking a

internal/core/fill.go

@@ -0,0 +1,59 @@
+package core
+
+import (
+	"encoding/binary"
+	// D11 promises the same bytes from the same seed, so a deliberate,
+	// reproducible generator is the product rather than a weakness. Nothing
+	// here ever makes a secret.
+	// nosemgrep: go.lang.security.audit.crypto.math_random.math-random-used
+	"math/rand/v2"
+)
+
+// Bulk random bytes are drawn eight at a time, and the two byte orders are two
+// functions rather than one function with a flag.
+//
+// Eight at a time rather than one is the whole point. Measured 2026-09-06 over
+// 64 MiB through a 32 KiB buffer, repetitions interleaved with the order
+// reversed: one draw per byte runs at 182 MB/s, eight bytes per draw at
+// 2499 MB/s. The write path of an archive is slower than the disk underneath it
+// when it draws a byte at a time (P2 in PERFORMANCE-REVIEW-2026-09-05.md).
+//
+// The orders stay separate because picking the wrong one is not a style
+// mistake - it silently rewrites every file a format has ever produced, which
+// is exactly what D11 forbids. A boolean argument would put that mistake one
+// typo away and would read the same in review either way. Two names cannot be
+// confused by accident, and the compiler cannot help with a flag.
+//
+// Neither function draws anything for an empty slice, and both spend exactly
+// one draw on a trailing group shorter than eight bytes. That is what the ten
+// call sites did before they were folded into here, so folding them in moved no
+// bytes except where the owner asked for them to move.
+
+// FillRandomBE fills b with random bytes, most significant byte of each draw
+// first. This is the order bmp, gif, ico, opc, png, tiff and - since 0.3.0 -
+// targz, wav and zip write.
+func FillRandomBE(b []byte, rng *rand.Rand) {
+	i := 0
+	for ; i+8 <= len(b); i += 8 {
+		binary.BigEndian.PutUint64(b[i:], rng.Uint64())
+	}
+	if i < len(b) {
+		var eight [8]byte
+		binary.BigEndian.PutUint64(eight[:], rng.Uint64())
+		copy(b[i:], eight[:])
+	}
+}
+
+// FillRandomLE fills b with random bytes, least significant byte of each draw
+// first. This is the order avif, jpg, jxl and webp write.
+func FillRandomLE(b []byte, rng *rand.Rand) {
+	i := 0
+	for ; i+8 <= len(b); i += 8 {
+		binary.LittleEndian.PutUint64(b[i:], rng.Uint64())
+	}
+	if i < len(b) {
+		var eight [8]byte
+		binary.LittleEndian.PutUint64(eight[:], rng.Uint64())
+		copy(b[i:], eight[:])
+	}
+}

internal/format/avif/avif.go

@@ -515,20 +515,11 @@ func writeFiller(ctx context.Context, w io.Writer, buf []byte, rng *rand.Rand, s
 			return ctx.Err()
 		default:
 		}
-		fillBytes(buf[:n], rng)
+		core.FillRandomLE(buf[:n], rng)
 		if _, err := w.Write(buf[:n]); err != nil {
 			return err
 		}
 		left -= n
 	}
 	return nil
 }
-
-func fillBytes(b []byte, rng *rand.Rand) {
-	for i := 0; i < len(b); i += 8 {
-		v := rng.Uint64()
-		for j := 0; j < 8 && i+j < len(b); j++ {
-			b[i+j] = byte(v >> (8 * uint(j)))
-		}
-	}
-}

internal/format/bmp/bmp.go

@@ -372,11 +372,7 @@ func writeGap(ctx context.Context, w io.Writer, seed uint64, n int64) error {
 			take = remaining
 		}
 		chunk := buf[:take]
-		for i := 0; i < len(chunk); i += 8 {
-			var eight [8]byte
-			binary.BigEndian.PutUint64(eight[:], rng.Uint64())
-			copy(chunk[i:], eight[:])
-		}
+		core.FillRandomBE(chunk, rng)
 		if _, err := w.Write(chunk); err != nil {
 			return err
 		}

internal/format/gif/gif.go

@@ -358,11 +358,7 @@ func writeComment(ctx context.Context, w io.Writer, seed uint64, blocks, payload
 		}
 		buf[0] = byte(size)
 		chunk := buf[1 : 1+size]
-		for j := 0; j < len(chunk); j += 8 {
-			var eight [8]byte
-			putUint64(eight[:], rng.Uint64())
-			copy(chunk[j:], eight[:])
-		}
+		core.FillRandomBE(chunk, rng)
 		if _, err := w.Write(buf[:1+size]); err != nil {
 			return err
 		}
@@ -372,17 +368,6 @@ func writeComment(ctx context.Context, w io.Writer, seed uint64, blocks, payload
 	return err
 }
 
-func putUint64(b []byte, v uint64) {
-	b[0] = byte(v >> 56)
-	b[1] = byte(v >> 48)
-	b[2] = byte(v >> 40)
-	b[3] = byte(v >> 32)
-	b[4] = byte(v >> 24)
-	b[5] = byte(v >> 16)
-	b[6] = byte(v >> 8)
-	b[7] = byte(v)
-}
-
 // sizeLadder is tried from the largest down when the recipe names no picture
 // size, exactly as PNG does. The first rung that leaves a reachable remainder
 // wins, so a small file gets a small picture instead of being refused.

internal/format/ico/ico.go

@@ -333,11 +333,7 @@ func writeGap(ctx context.Context, w io.Writer, seed uint64, n int64) error {
 			take = remaining
 		}
 		chunk := buf[:take]
-		for i := 0; i < len(chunk); i += 8 {
-			var eight [8]byte
-			binary.BigEndian.PutUint64(eight[:], rng.Uint64())
-			copy(chunk[i:], eight[:])
-		}
+		core.FillRandomBE(chunk, rng)
 		if _, err := w.Write(chunk); err != nil {
 			return err
 		}

internal/format/jpg/jpg.go

@@ -543,14 +543,7 @@ func writeComments(ctx context.Context, w io.Writer, seed uint64, segments int,
 				size = remaining
 			}
 			chunk := buf[:size]
-			for j := 0; j < len(chunk); j += 8 {
-				var eight [8]byte
-				v := rng.Uint64()
-				for k := 0; k < 8; k++ {
-					eight[k] = byte(v >> (8 * k))
-				}
-				copy(chunk[j:], eight[:])
-			}
+			core.FillRandomLE(chunk, rng)
 			if _, err := w.Write(chunk); err != nil {
 				return err
 			}

internal/format/jxl/jxl.go

@@ -566,20 +566,11 @@ func writeFiller(ctx context.Context, w io.Writer, buf []byte, rng *rand.Rand, s
 			return ctx.Err()
 		default:
 		}
-		fillBytes(buf[:n], rng)
+		core.FillRandomLE(buf[:n], rng)
 		if _, err := w.Write(buf[:n]); err != nil {
 			return err
 		}
 		left -= n
 	}
 	return nil
 }
-
-func fillBytes(b []byte, rng *rand.Rand) {
-	for i := 0; i < len(b); i += 8 {
-		v := rng.Uint64()
-		for j := 0; j < 8 && i+j < len(b); j++ {
-			b[i+j] = byte(v >> (8 * uint(j)))
-		}
-	}
-}

internal/format/opc/opc.go

@@ -24,7 +24,6 @@ import (
 	"bytes"
 	"compress/flate"
 	"context"
-	"encoding/binary"
 	"fmt"
 	"hash/crc32"
 	"io"
@@ -412,11 +411,7 @@ func writeFiller(ctx context.Context, w io.Writer, seed uint64, n int64) error {
 			take = remaining
 		}
 		chunk := buf[:take]
-		for i := 0; i < len(chunk); i += 8 {
-			var eight [8]byte
-			binary.BigEndian.PutUint64(eight[:], rng.Uint64())
-			copy(chunk[i:], eight[:])
-		}
+		core.FillRandomBE(chunk, rng)
 		if _, err := w.Write(chunk); err != nil {
 			return err
 		}

internal/format/png/png.go

@@ -470,11 +470,7 @@ func writePaddingChunk(ctx context.Context, w io.Writer, kind string, seed uint6
 			size = remaining
 		}
 		chunk := buf[:size]
-		for i := 0; i < len(chunk); i += 8 {
-			var eight [8]byte
-			binary.BigEndian.PutUint64(eight[:], rng.Uint64())
-			copy(chunk[i:], eight[:])
-		}
+		core.FillRandomBE(chunk, rng)
 
 		crc.Write(chunk)
 		if _, err := w.Write(chunk); err != nil {