BC5 readiness: pairwise BC4↔BC5 + check-bc5-compat orchestrator + scheduled CI

.github/workflows/live-canary.yml

@@ -0,0 +1,142 @@
+name: Live Canary
+
+# Runs the BC4↔BC5 live compatibility canary against real Basecamp backends.
+#
+# Triggers
+# --------
+# - workflow_dispatch: manual run, useful for ad-hoc verification.
+# - schedule: nightly cron at 09:00 UTC. Catches server-side drift between
+#   the BC4 (master) baseline and the BC5 (five) branch over time.
+#
+# Required secrets (configured at repo level)
+# -------------------------------------------
+#   BASECAMP_TOKEN       OAuth token for both backends. Must have read scope
+#                        on the canary's read-only fixture set.
+#   BASECAMP_ACCOUNT_ID  The numeric account ID used to compose the per-backend
+#                        base URL. Same on both backends — the canary requires
+#                        identical account state across BC4 and BC5 for
+#                        pairwise rules to be meaningful.
+#   BC5_HOST             Origin of the BC5 backend (e.g. https://5.basecampapi.com).
+#
+# Optional secret (defaults)
+# --------------------------
+#   BASECAMP_HOST        Origin of the BC4 backend. Defaults to
+#                        https://3.basecampapi.com when unset.
+#
+# The job no-ops with a clear log message if BASECAMP_TOKEN is unset (e.g.
+# when running from a fork without secrets). It does not fail the run —
+# missing secrets are operator-controllable state.
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '0 9 * * *'
+
+permissions: {}
+
+concurrency:
+  group: live-canary
+  cancel-in-progress: false
+
+jobs:
+  canary:
+    name: BC4↔BC5 canary
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+
+      - name: Check for canary secrets
+        id: secrets
+        env:
+          BASECAMP_TOKEN: ${{ secrets.BASECAMP_TOKEN }}
+          BASECAMP_ACCOUNT_ID: ${{ secrets.BASECAMP_ACCOUNT_ID }}
+          BC5_HOST: ${{ secrets.BC5_HOST }}
+        run: |
+          if [ -z "$BASECAMP_TOKEN" ] || [ -z "$BASECAMP_ACCOUNT_ID" ] || [ -z "$BC5_HOST" ]; then
+            echo "Canary secrets not configured; skipping live run."
+            echo "Required: BASECAMP_TOKEN, BASECAMP_ACCOUNT_ID, BC5_HOST."
+            echo "configured=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "configured=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Set up Go
+        if: steps.secrets.outputs.configured == 'true'
+        uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
+        with:
+          go-version-file: 'go/go.mod'
+          cache-dependency-path: 'go/go.sum'
+
+      - name: Set up Node.js
+        if: steps.secrets.outputs.configured == 'true'
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        with:
+          node-version: '22'
+          cache: 'npm'
+          cache-dependency-path: |
+            typescript/package-lock.json
+            conformance/runner/typescript/package-lock.json
+
+      - name: Set up Java
+        if: steps.secrets.outputs.configured == 'true'
+        uses: actions/setup-java@1bcf9fb12cf4aa7d266a90ae39939e61372fe520 # v5.4.0
+        with:
+          distribution: 'temurin'
+          java-version: '17'
+
+      - name: Setup Gradle
+        if: steps.secrets.outputs.configured == 'true'
+        uses: gradle/actions/setup-gradle@3f131e8634966bd73d06cc69884922b02e6faf92 # v6.2.0
+        with:
+          cache-provider: basic
+
+      - name: Set up Ruby
+        if: steps.secrets.outputs.configured == 'true'
+        uses: ruby/setup-ruby@d45b1a4e94b71acab930e56e79c6aa188764e7f9 # v1.316.0
+        with:
+          ruby-version: '3.3'
+          bundler-cache: true
+          working-directory: conformance/runner/ruby
+
+      - name: Install uv (Python)
+        if: steps.secrets.outputs.configured == 'true'
+        uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
+
+      - name: Build TypeScript SDK
+        if: steps.secrets.outputs.configured == 'true'
+        working-directory: typescript
+        run: |
+          npm ci
+          npm run build
+
+      - name: Run live canary against BC4 + BC5 with pairwise comparison
+        if: steps.secrets.outputs.configured == 'true'
+        env:
+          BASECAMP_TOKEN: ${{ secrets.BASECAMP_TOKEN }}
+          BASECAMP_ACCOUNT_ID: ${{ secrets.BASECAMP_ACCOUNT_ID }}
+          # Default the BC4 origin here: an unset optional secret expands to ''.
+          # The Makefile's recipe-shell ${BASECAMP_HOST:-...} default would also
+          # catch the empty string, but defaulting explicitly keeps the workflow
+          # self-describing about which BC4 origin a run used.
+          BASECAMP_HOST: ${{ secrets.BASECAMP_HOST || 'https://3.basecampapi.com' }}
+          BC5_HOST: ${{ secrets.BC5_HOST }}
+          LIVE_RECORD_DIR: tmp/live-canary
+        run: make check-bc5-compat
+
+      # Deliberately EXCLUDES the wire/ trees: wire snapshots carry raw
+      # response bodies and captured headers from a live account, which must
+      # not persist as downloadable workflow artifacts. Decode results carry
+      # only booleans, error strings, and dotted field paths.
+      - name: Upload canary decode results
+        if: always() && steps.secrets.outputs.configured == 'true'
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: live-canary-${{ github.run_id }}
+          path: |
+            tmp/live-canary/*/decode/
+          retention-days: 14
+          if-no-files-found: warn

CONTRIBUTING.md

@@ -15,6 +15,10 @@ Thank you for your interest in contributing to the Basecamp SDK. This document p
 | Kotlin | JDK 17+, Kotlin 2.0+ |
 | Python | Python 3.11+, [uv](https://docs.astral.sh/uv/) |
 
+Shared tooling: `jq`, and bash >= 4.4 on `PATH` for the pairwise-canary
+scripts that `make check` runs (macOS ships bash 3.2 at `/bin/bash` —
+`brew install bash`; the scripts fail fast with this exact hint).
+
 A Basecamp account is optional (for integration testing only).
 
 ### Getting Started
@@ -275,8 +279,9 @@ Optional env:
 - `BASECAMP_BACKEND=bc4|bc5` — namespaces persisted snapshots so BC4 and
   BC5 runs don't collide.
 - `LIVE_RECORD_DIR=<path>` — persists wire snapshots to
-  `<path>/<backend>/wire/<test>.json`. Used by downstream cross-language
-  decoders (PR 3) and BC4↔BC5 comparison (PR 4).
+  `<path>/<backend>/wire/<test>.json`. Consumed by the cross-language
+  replay runners (`make conformance-*-replay`) and by the pairwise
+  BC4↔BC5 comparison (`scripts/compare-canary-runs.sh`).
 - `BASECAMP_BC4_PROJECT_ID` / `BASECAMP_BC5_PROJECT_ID` /
   `BASECAMP_PROJECT_ID` etc. — explicit fixture-IDs override the runner's
   discovery walk. Same pattern applies for `TODOSET_ID`, `TODOLIST_ID`,
@@ -348,11 +353,6 @@ hand-rolled schema walkers (which mirror the TS validator's required + extras
 algorithm in each language). Any per-language divergence in `extras_seen`
 points at a walker bug in the diverging language.
 
-`make conformance-typescript-live` is owned by PR 2; the four
-`make conformance-*-replay` targets ship in PR 3. The orchestrating
-`make conformance-live` and `make check-bc5-compat` targets that thread
-TS capture → replay → BC4↔BC5 comparison together land in PR 4.
-
 When the SDK gains a new operation in `live-my-surface.json`, it must be
 added to:
 
@@ -364,6 +364,123 @@ added to:
 
 Each runner's coverage gate refuses to start until all five are in place.
 
+### Pairwise BC4↔BC5 comparison
+
+Per-backend schema validation is necessary but not sufficient. With every
+new BC5 field marked optional, a regression where BC4 emits `memories:
+[item, item]` and BC5 emits `memories: []` would pass per-backend schema
+checks — yet that's exactly the additive-only invariant the canary should
+catch. The pairwise comparison closes that loop.
+
+Each live test in `conformance/tests/live-my-surface.json` can carry
+`pairwiseAssertions`, which apply to a matched pair of BC4 and BC5 wire
+snapshots for the same operation. Four rule types:
+
+- `pairwiseSupersetArray` — BC5 array length at each path must be ≥ BC4's.
+  Catches "this array shrank between backends".
+- `pairwiseSupersetKeys` — BC5 object's keys at each path must be a superset
+  of BC4's keys. Catches "this field disappeared".
+- `pairwiseEqual` — BC5 value at each path must equal BC4's. Use sparingly;
+  most useful for type-discriminator fields.
+- `pairwiseDeltaAllowed` — paths where divergence is explicitly accepted.
+  The listed paths are skipped by the other three rules for this operation.
+  `reason` is **required** — a public compatibility report indexes accepted
+  divergences.
+
+Path syntax (dotted identifiers, evaluated against each snapshot):
+
+- `""` (empty string) addresses the body root.
+- `foo.bar` is shorthand for `pages[0].body.foo.bar` — single-page bodies.
+- `pages[N].body.X` addresses a specific page in multi-page snapshots.
+- `pages[*].body.X` aggregates across pages into a list (each page's value
+  becomes one element of the result). `pairwiseEqual` compares that list as
+  a single value. `pairwiseSupersetArray` compares the **total item count
+  across pages** (a page missing the field contributes 0; any non-null
+  non-array page value is invalid), so a field dropped on every page fails
+  even when page counts match, and redistribution across page boundaries
+  with the total preserved passes. `pairwiseSupersetKeys` does **not**
+  support `pages[*]` — the aggregate is a list, not an object, and the rule
+  flags it as an invalid target; address a specific page
+  (`pages[N].body.X`) for keys checks on paginated endpoints.
+
+Example — the canonical `memories` canary on `GetMyNotifications`:
+
+```json
+"pairwiseAssertions": [
+  {
+    "type": "pairwiseSupersetArray",
+    "paths": ["memories"],
+    "reason": "Additive-only invariant: BC5 memories[] should remain a superset of BC4's. PERMANENTLY waived by the pairwiseDeltaAllowed entry below — bc3 documents memories as an always-empty placeholder on BC5 (doc/api/sections/my_notifications.md). See spec/api-gaps/memories-emptied-regression.md."
+  },
+  {
+    "type": "pairwiseDeltaAllowed",
+    "paths": ["memories"],
+    "reason": "PERMANENT documented contract divergence: BC5 ships `json.memories []` by contract — doc/api/sections/my_notifications.md (bc3 #11628) codifies memories as an always-empty placeholder superseded by bubble_ups. The once-planned alias (`json.memories @bubble_ups`, bc3 #10947) never shipped; #10947 is closed unmerged. Not a pending regression — this entry is the machine-readable record of an accepted BC4→BC5 subtractive delta. Retire only if BC4 (the four branch) empties memories too (the superset rule then passes clean) or BC5's documented contract changes. Tracked in spec/api-gaps/memories-emptied-regression.md."
+  }
+]
+```
+
+BC5 ships `memories: []` permanently by documented contract —
+`doc/api/sections/my_notifications.md` (bc3 #11628) codifies memories as an
+always-empty placeholder superseded by `bubble_ups`. The superset rule stays
+as the encoded additive-only invariant; the `pairwiseDeltaAllowed` entry is
+the machine-readable record of the accepted delta, feeding the public
+compatibility report. Retire the waiver only if BC4 (the four branch) empties
+memories too — the superset rule then passes clean — or BC5's documented
+contract changes.
+
+### Orchestrator
+
+`make check-bc5-compat` threads the two-backend capture plus pairwise
+comparison together:
+
+```bash
+BASECAMP_TOKEN=<token> \
+BASECAMP_ACCOUNT_ID=<account> \
+BC5_HOST=https://5.basecampapi.com \
+make check-bc5-compat
+```
+
+What it runs, in order:
+
+1. `BASECAMP_BACKEND=bc4 LIVE_RECORD_DIR=tmp/live-canary make conformance-live`
+   — TS captures wire snapshots from BC4 (defaulting `BASECAMP_HOST` to
+   `https://3.basecampapi.com`), then Ruby/Python/Go/Kotlin replay-decode.
+2. `BASECAMP_BACKEND=bc5 LIVE_RECORD_DIR=tmp/live-canary make conformance-live`
+   — same against the BC5 origin set via `BC5_HOST`.
+3. `./scripts/compare-canary-runs.sh tmp/live-canary/bc4/wire tmp/live-canary/bc5/wire`
+   — applies pairwise rules. Reports all violations outside
+   `pairwiseDeltaAllowed` and exits non-zero if any are found.
+
+Override `LIVE_RECORD_DIR` (default `tmp/live-canary`) or `BASECAMP_HOST`
+(default `https://3.basecampapi.com`) on the make line.
+
+**Identical account state across both runs is mandatory.** The pairwise
+rules assume the BC4 and BC5 backends see the same projects, the same
+todosets, the same reading list, etc. Without that, `unreads`/`reads`
+arrays drift naturally between captures and pairwise asserts will false-fail.
+Use a dedicated test account with stable, equivalent fixtures (e.g. a
+sandbox account snapshot replicated to each backend).
+
+### Scheduled CI
+
+`.github/workflows/live-canary.yml` runs `check-bc5-compat` nightly via
+cron and on `workflow_dispatch`. It is **opt-in**: the workflow no-ops
+with a clear log message if the required repo secrets aren't configured.
+
+Required secrets:
+
+- `BASECAMP_TOKEN` — OAuth token with read scope for the canary fixtures.
+- `BASECAMP_ACCOUNT_ID` — the numeric account ID used on both backends.
+- `BC5_HOST` — origin of the BC5 backend.
+
+Optional:
+
+- `BASECAMP_HOST` — origin of the BC4 backend; defaults to `https://3.basecampapi.com`.
+
+Snapshots are uploaded as a workflow artifact (`live-canary-<run-id>`,
+14-day retention) so failures can be inspected post-hoc without rerunning.
+
 ## API gap registry (`spec/api-gaps/`)
 
 When BC ships a new user-visible feature without a JSON API (or with an

COORDINATION.md

@@ -68,7 +68,6 @@ Both records live on in the registry and this SDK plan's §8.
   `five+api`)
 - SDK-side absorption status: [`spec/api-gaps/README.md`](spec/api-gaps/README.md)
 - Live canary results: `make check-bc5-compat` against current refs
-  (lands in PR 4 of the SDK plan)
 - Cross-team alignment status: this file
 
 — The basecamp-sdk team