Add processing pipeline test spec with golden outputs

.github/workflows/test.yml

@@ -97,6 +97,16 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v4
+      - name: Pull Tesseract LFS libraries
+        # Only the OCR libraries, so the processing spec tests can run OCR
+        # cases; a full LFS checkout would also fetch the large LibreOffice
+        # bundle, which the tests do not need
+        run: git lfs pull --include "documentcloud/documents/processing/ocr/tesseract/*"
+      - name: Cache OCR language data
+        uses: actions/cache@v4
+        with:
+          path: documentcloud/documents/processing/tests/spec/.cache
+          key: processing-spec-tessdata-7d4322bd
       - name: Install Python
         uses: actions/setup-python@v5
         with:
@@ -112,3 +122,6 @@ jobs:
           PG_USER: test
           PG_PASSWORD: ${{ secrets.PG_PASSWORD }}
           DATABASE_URL: postgres://test:postgres@127.0.0.1:5432/test
+          # OCR is expected to work in CI: fail the processing spec's OCR
+          # cases instead of skipping them if the runtime is unavailable
+          DC_SPEC_REQUIRE_OCR: "1"

documentcloud/documents/processing/tests/spec/.gitignore

@@ -0,0 +1 @@
+.cache/

documentcloud/documents/processing/tests/spec/README.md

@@ -0,0 +1,216 @@
+# Processing Pipeline Test Spec
+
+Golden-output tests for the document processing pipeline. For each test
+document in `documents/`, the `expected/` directory contains **every file the
+current pipeline produces** for that document, plus the database-facing
+metadata sent back to the API. The goal is to be able to change processing
+code and validate that the outputs (or exactly the outputs you intended to
+change) stay the same.
+
+This builds on the output-contract spec in
+[MuckRock/research `processing-pipeline-spec`](https://github.com/MuckRock/research/tree/main/processing-pipeline-spec),
+which documents *what* the pipeline must produce. This directory makes that
+contract executable against the real implementation.
+
+## Layout
+
+```
+spec/
+├── harness.py          # runs the real pipeline in-process (no mocks)
+├── compare.py          # comparison / normalization rules
+├── generate.py         # regenerates the golden outputs
+├── test_spec.py        # pytest: fresh run must match expected/
+├── make_corpus.py      # builds the input PDFs (deterministic)
+└── documents/
+    └── <case>/
+        ├── case.json   # doc_id, slug, processing settings, redactions
+        ├── input/      # the "uploaded" document
+        └── expected/   # everything the pipeline produced
+            ├── <slug>.pdf         # processed PDF (OCR text grafted in)
+            ├── <slug>.txt         # concatenated text, pages joined by \n\n
+            ├── <slug>.txt.json    # per-page text + OCR metadata
+            ├── <slug>.index       # I/O cache (implementation artifact)
+            ├── pages/
+            │   ├── <slug>-p<N>-{xlarge,large,normal,small,thumbnail}.gif
+            │   ├── <slug>-p<N>.txt
+            │   └── <slug>-p<N>.position.json
+            └── metadata.json      # captured API callbacks: page_count,
+                                   # page_spec, file_hash, status
+```
+
+`metadata.json` is not a pipeline storage file — it records what the pipeline
+PATCHed back to the API (the `database` key is the merged final state, and
+`callbacks` is the raw sequence of requests).
+
+## How the harness works
+
+`harness.PipelineRunner` runs the actual serverless functions
+(`info_and_image/main.py`, `ocr/main.py`) with:
+
+- the `local` storage environment writing under a temporary `MEDIA_ROOT`
+- a real Redis (`REDIS_PROCESSING_URL`, defaults to `localhost:6379` — the
+  same requirement the existing pipeline tests have in CI)
+- pubsub topics dispatched through an in-process FIFO queue, so each function
+  runs to completion before the next starts (in production every message runs
+  in its own Lambda; pdfium cannot be nested inside one process)
+- API callbacks (`serverless.utils.request`) captured instead of sent
+
+Nothing inside the pipeline is mocked: pdfium rendering and text extraction,
+Tesseract OCR, PDF grafting, pdfplumber text positions, and page_spec
+crunching all run for real.
+
+## Test corpus
+
+| Case | Pages | What it validates |
+| --- | --- | --- |
+| `text-1page` | 1 | Baseline: embedded text, no OCR, full output file set |
+| `text-3page` | 3 | All pages get all artifacts; concatenation order |
+| `text-4page` | 4 | More pages than `TEXT_POSITION_BATCH` (3): multi-batch flush path |
+| `scan-2page` | 2 | Image-only pages: OCR text, grafting, OCR positions |
+| `mixed-2page` | 2 | Per-page routing: page 1 `ocr: null`, page 2 `ocr: "tess4"` |
+| `force-ocr-1page` | 1 | `force_ocr` overrides embedded text (`ocr: "tess4_force"`) |
+| `mixed-sizes-2page` | 2 | Differing page dimensions; page_spec encoding |
+| `blank-1page` | 1 | Blank page is OCR'd; produces `"\f"` text and `[]` positions |
+| `redact-2page` | 2 | Redaction path: only the dirty page is reprocessed; page_spec is not resent |
+| `corrupt-1page` | — | Error path: an unparseable PDF produces an error POST to the API and no output files |
+
+Inputs are built deterministically by `make_corpus.py` and committed, so the
+goldens always correspond to known input bytes.
+
+## Running the validation
+
+```bash
+pytest documentcloud/documents/processing/tests/spec/test_spec.py
+```
+
+This runs in CI alongside the other tests (it needs the Redis service the
+existing pipeline tests already use). The CI workflow pulls the Tesseract
+LFS libraries so the OCR cases run there too; in a local checkout without
+them, OCR cases are skipped and the embedded-text cases still run. The
+pinned eng.traineddata is downloaded automatically on first use.
+
+To check outside of pytest:
+
+```bash
+python documentcloud/documents/processing/tests/spec/generate.py --check
+```
+
+## Regenerating the goldens
+
+After an *intentional* change to pipeline behavior:
+
+```bash
+python documentcloud/documents/processing/tests/spec/generate.py [case ...]
+```
+
+then review the diff — it is a precise inventory of what your change did to
+the pipeline's output — and commit it.
+
+### OCR requirements
+
+The OCR cases run the bundled Tesseract, which is stored in Git LFS:
+
+```bash
+git lfs pull --include "documentcloud/documents/processing/ocr/tesseract/*"
+```
+
+OCR output is only comparable when produced with the same language data, so
+the harness pins `eng.traineddata` by content hash
+(`tessdata_fast` 4.1.0; `generate.py` downloads it into `.cache/` on first
+use). If you want goldens that match production OCR exactly, place the
+production `eng.traineddata` (from the `ocr-languages` storage folder) in
+`.cache/` instead and regenerate — but note the committed goldens must then
+be produced with that same file.
+
+## Validating a modified or replacement pipeline
+
+The corpus and comparison logic are deliberately independent of the current
+implementation: `documents/` is a set of input → expected-output pairs, and
+`compare.compare_case(expected_dir, actual_dir)` only looks at two
+directories. `harness.py` is the *current* pipeline's runner; to validate a
+replacement, write a runner for it that stages each case's input, produces a
+document directory in the same layout, and records the final database
+metadata (plus any error reports) in `metadata.json` — then compare with
+`strictness="equivalent"`.
+
+Two strictness modes:
+
+- **`exact`** (default, used by `test_spec.py` and CI) answers "is the
+  current pipeline unchanged?" — byte-exact text and images, full API
+  callback sequence. This intentionally pins implementation details,
+  including quirks like pdfium's trailing NUL and Tesseract's trailing form
+  feed, so any change is visible.
+- **`equivalent`** (`generate.py --check --equivalent`, or
+  `strictness="equivalent"`) answers "does a different implementation
+  produce equivalent final output?" — trailing control characters are
+  normalized away, OCR text is held to a similarity threshold instead of
+  equality, images to a mean pixel delta instead of identical bytes, the
+  `ocr` field to OCR'd-or-not instead of an engine name, and the callback
+  sequence is not compared (though error reports must still be sent). The
+  thresholds at the top of `compare.py` are a first cut — tune them against
+  real replacement candidates.
+
+Open question for the team: which of the pinned quirks are actually
+depended on downstream (search indexing, the frontend, add-ons)? The
+equivalence rules currently treat the trailing NUL/form-feed as accidents a
+replacement is free to fix; if a consumer turns out to rely on one, move it
+into the contract and tighten the rule.
+
+## Comparison rules (exact mode)
+
+Some pipeline output is intentionally or unavoidably non-reproducible, so
+`compare.py` normalizes even in exact mode:
+
+| Output | Rule |
+| --- | --- |
+| `*.txt` (full and per-page) | exact bytes |
+| `*.txt.json` | JSON equality; `updated` timestamps ignored (wall clock) |
+| `*.position.json` | JSON equality; coordinate floats within `1e-4` |
+| `*.gif` | exact bytes; on mismatch, pixels are diffed and reported |
+| `*.pdf` | semantic: page count, page dimensions, per-page text layer. Bytes are not compared — the OCR grafter writes uuid-named XObjects and pikepdf regenerates the document ID |
+| `*.index` | presence only (gzip bytes embed a timestamp; not part of the contract) |
+| `metadata.json` | the merged database fields **and** the full API callback sequence (order, methods, URLs, payloads). `page_count` and `status` exact; `page_spec` compared decoded (segment order follows Redis set iteration and is not deterministic); `file_hash` exact, except for redaction cases where the final PDF is rewritten non-reproducibly — there the callbacks still assert that a well-formed SHA-1 was sent, just not its value. Storage bucket prefixes in error messages are stripped (the bucket name varies by environment) |
+
+Behaviors of the current pipeline that the goldens capture and that any
+reimplementation would need to match (or knowingly change):
+
+- Text extracted by pdfium ends with a trailing NUL (`\x00`) character
+- Text produced by Tesseract ends with a form feed (`\f`); a blank page
+  yields `"\f"`, not the empty string
+- Pages in `<slug>.txt` are joined with `\n\n`
+- `ocr` values in `txt.json`: `null` (embedded text), `"tess4"`,
+  `"tess4_force"` (when `force_ocr` is set)
+- The `updated` timestamps are milliseconds; each page carries its own
+- `file_hash` is the SHA-1 of the PDF as hashed during page-cache processing
+  (before OCR grafting rewrites it)
+- Redaction re-sends `file_hash` and `status` but not `page_spec`
+
+## Not covered yet
+
+- **Textract OCR** (`ocr_engine: "textract"`) — requires AWS and an AI-credit
+  API; the golden harness only runs Tesseract
+- **Non-PDF uploads** — document conversion needs the LibreOffice bundle
+  (`document_conversion/libreoffice/lo.tar.gz`, a large LFS object)
+- **Page modifications** (reorder / rotate / insert) — the modify path needs
+  `storage.async_download`, which the local storage backend does not implement
+- **Bulk import**, **set_page_text**, revision control copies
+- **Non-English OCR** — needs additional pinned traineddata files
+- **Large documents** — every case fits in a single `IMAGE_BATCH` (55 pages);
+  multi-batch image extraction is not exercised (text-position batching is,
+  via `text-4page`, and OCR batching via `scan-2page` since `OCR_BATCH` is 1)
+- **Access levels** — all cases run with `access: private`; local storage
+  ignores ACLs, so a regression in how `access` propagates between pipeline
+  messages would not be caught
+- **Error paths beyond an unparseable input** (`corrupt-1page`) — e.g.
+  OCR failures, storage failures, and the retry queue
+- **Real-world documents** — the corpus is synthetic (built with PyMuPDF),
+  so rotated pages, non-Latin/RTL text, ligatures, skewed scans, forms,
+  broken xref tables, JPEG2000 images, and unusual encodings are not
+  represented. Adding a handful of small, redistributable real documents is
+  the highest-value corpus improvement for replacement confidence
+
+Platform note: the GIF goldens are byte-compared and therefore pinned to the
+Linux builds of pdfium/Pillow used in CI. Run the suite on Linux (or in the
+dev container); a macOS run will report pixel deltas, and the bundled
+Tesseract libraries only load on Linux x86-64 (OCR cases skip elsewhere —
+or fail if `DC_SPEC_REQUIRE_OCR` is set, as it is in CI).