Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions .agents/checks/review.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ Checks for AI review agents. The authoritative rules live in [SAFETY.md](../../S
[AGENTS.md](../../AGENTS.md), and [docs/tcb-model.md](../../docs/tcb-model.md) — this file is
the reviewer's distillation.

- Judge the change through both project lenses: (a) OSS-first — pg-sprite as the preferred
standalone PostgreSQL online-DDL tool (CLI usable without an orchestrator, external-user
docs and errors, no Block-internal assumptions); (b) clean SchemaBot integration — a stable
adapter-friendly seam (library API, verdict/plan JSON, error taxonomy) with the core never
depending on SchemaBot. Flag changes that serve one lens at the other's expense without a
recorded decision.
- Look up every touched `pkg/` package in the SAFETY.md partition table first — the review bar
differs between the safety-critical core and the periphery. Flag core changes with 🌶️ and
state the blast radius (data corruption, lost writes, wrong-table swap, stranded slot).
Expand All @@ -21,8 +27,11 @@ the reviewer's distillation.
module — ideas are ported with citations, not code.
- Connections go through `pkg/dbconn` (bounded `lock_timeout` / `statement_timeout`) — flag
raw `pgx` pools in production code.
- SQL parsing goes through `pg_query_go`; flag `strings.Split(";")` or any hand-parsing. A
parse failure is an error surfaced to the caller.
- SQL parsing goes through `wasilibs/go-pgquery` (Wasm `libpg_query`); flag
`strings.Split(";")`, any hand-parsing, and imports of the cgo `pg_query_go` (documented
escape hatch, not the default). A parse failure is an error surfaced to the caller.
Shadow-table DDL and checkpoint fingerprints come from execute-and-introspect on the scratch
database — flag AST surgery that constructs the shadow schema or fingerprints SQL text.
- Generated SQL quotes every user-supplied or introspected identifier
(`pgx.Identifier{...}.Sanitize()` / `quote_ident()`) — flag raw interpolation of names into
SQL. Connection strings are parsed and re-serialized (`pgx.ParseConfig`), never
Expand Down
8 changes: 8 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,14 @@ jobs:
with:
go-version-file: go.mod
- run: make build
# The no-cgo promise is a contract: the Wasm parser keeps `go install`
# toolchain-free. One accidental import of the cgo escape hatch
# (pg_query_go's parser) would silently start requiring a C toolchain
# on every contributor's machine.
- name: Build with CGO disabled
run: go build ./...
env:
CGO_ENABLED: "0"

# The integration suite runs against every Aurora-supported PostgreSQL
# major (see docs/postgresql-version-support.md): the version floor is a
Expand Down
16 changes: 16 additions & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,16 @@ linters:
- gochecknoinits # no init() functions
- gochecknoglobals # no package-level mutable state (error sentinels exempt)
- containedctx # no context.Context stored in struct fields
- sloglint # structured-logging hygiene: static messages, snake_case keys
- forbidigo # no printing to process stdout; output goes to the injected writer
settings:
sloglint:
static-msg: true
key-naming-case: snake
forbidigo:
forbid:
- pattern: ^(fmt\.Print(f|ln)?|print|println)$
msg: command output goes to the injected io.Writer and diagnostics to slog, never process stdout
usetesting:
context-background: true
context-todo: true
Expand All @@ -32,6 +41,13 @@ linters:
- name: package-comments
exclusions:
rules:
# The tracelog adapter forwards pgx's own message strings; the
# static-message rule applies to messages we author, not to a bridge
# for a foreign logger.
- path: pkg/dbconn/dbconn\.go
linters:
- sloglint
text: message should be a string literal
# Test helpers intentionally keep uniform signatures and fixed arguments
# for readability, so unused/constant params there are not worth churn.
- path: _test\.go
Expand Down
50 changes: 47 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,18 @@ This file is canonical. `CLAUDE.md`, `GEMINI.md`, `.cursorrules`, `.goosehints`,
`.github/copilot-instructions.md` are symlinks to it — edit only this file. Review-agent
checks live in [.agents/checks/review.md](.agents/checks/review.md).

## Two lenses on every change

Judge every PR, design, and review through both lenses — a change that serves one at the
expense of the other needs an explicit decision, not a silent trade:

1. **OSS-first.** pg-sprite aims to be the preferred PostgreSQL online-DDL tool in its own
right: the CLI works standalone with no orchestrator setup, docs and error messages are
written for external users, and nothing assumes a Block-internal environment.
2. **Clean SchemaBot integration.** pg-sprite must slot into SchemaBot as an engine behind a
stable seam: keep the library API, verdict/plan JSON contracts, and error taxonomy
adapter-friendly, and never let the core depend on SchemaBot (or any orchestrator).

## Read SAFETY.md first

This codebase is partitioned into a **safety-critical core** and a periphery.
Expand Down Expand Up @@ -51,8 +63,12 @@ make lint # golangci-lint
generated SQL goes through `pgx.Identifier{...}.Sanitize()` (or `quote_ident()` server-side).
- Never string-manipulate connection strings/DSNs — parse (`pgx.ParseConfig`), modify fields,
re-serialize; string ops break on passwords containing `/`, `@`, or `%`.
- All SQL parsing goes through `pg_query_go` (once `pkg/statement` exists). No
`strings.Split(";")`, no hand-parsing; a parse failure is an error surfaced to the caller.
- All SQL parsing goes through the real PostgreSQL grammar via `wasilibs/go-pgquery` (Wasm
`libpg_query`; the cgo `pg_query_go` is the API-compatible escape hatch, not the default),
with `pkg/statement` as the parse boundary. No `strings.Split(";")`, no hand-parsing; a parse failure is an
error surfaced to the caller. Shadow-table DDL and checkpoint fingerprints are derived by
execute-and-introspect on the engine-owned scratch database, never by AST transformation
(see [docs/low-level-design.md](docs/low-level-design.md#how-the-planner-understands-ddl-decided)).
- Tests use testify (`require` for setup, `assert` for verification), `t.Context()` (in
cleanups, which run after the context is cancelled, use
`context.WithoutCancel(t.Context())`), and named polling deadlines — no bare `time.Sleep`
Expand Down Expand Up @@ -104,8 +120,36 @@ make lint # golangci-lint
- Never reference internal company details (cluster names, hostnames, org names) in code,
comments, commits, or PRs — this is a public repo.

## Logging and observability

- **stdout is the product's output; diagnostics go to stderr.** Command results (verdicts,
status) are written to the injected writer only; everything diagnostic goes through
`log/slog`. `--debug` on DB commands enables statement-level tracing (pgx tracelog via
`pkg/dbconn`) plus lifecycle events; without it, diagnostics are discarded.
- **Log decisions and state transitions, not progress noise.** Static messages; the
variability goes into attrs with stable snake_case keys, and the same key means the same
thing everywhere (`schema`, `table`, `total_bytes`, `elapsed`).
- **Logs answer the triage question.** Error- and warn-path logs carry the identifiers an
operator needs to act — schema, table, database, the operation being attempted — as
attrs, not buried in prose.
- **One error, one log.** Errors are wrapped and returned; only the entry point logs or
prints them. `pkg/` packages never log an error they also return.
- **Never log credentials or connection strings** — a DSN/URL carries a password; log host,
database, and user as separate attrs when needed. Never log row data.
- **Log output is never a test surface.** Tests assert typed outcomes — `errors.Is`/`As`,
verdict fields, exit codes, JSON output — never log text or human-facing wording. If a
behavioral difference is visible only in prose, make it machine-readable first (a typed
field or reason), then test that. The only exception is a renderer's own unit test.
- **Operational quantities ride on logs until there is a metrics runtime.** Durations,
sizes, and retry counts are logged as attrs. When the long-running phases need real
metrics, they arrive as OpenTelemetry instruments behind one engine-owned `pkg/metrics`
with `Record*` helpers — dotted `pgsprite.` names with explicit units, low-cardinality
snake_case attributes, counters for rare or dangerous branches operators can act on —
never direct exporter imports in core (the dependency rule in SAFETY.md applies).

Mechanical style rules (doc comments on exported symbols, no `init()`, no package-level
mutable state, no `context.Context` in structs) are enforced by `.golangci.yml`, not prose.
mutable state, no `context.Context` in structs, static slog messages with snake_case keys,
no printing to process stdout) are enforced by `.golangci.yml`, not prose.

## Git and PRs

Expand Down
11 changes: 8 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,14 @@ when one exists (`CONCURRENTLY`, `NOT VALID` + `VALIDATE`, fast default,
`USING INDEX`), and a log-based, checksum-gated, resumable copy-and-swap when
a genuine table rewrite is unavoidable.

**Status: Phase 0 (scaffold + test harness).** All subcommands are stubs. The
design docs and the phased build plan live in [docs/](docs/) — start with
[docs/README.md](docs/README.md).
**Status: Phase 1 (optimistic front door).** `pg-sprite migrate --alter '…'`
runs easy `ALTER TABLE` changes directly under tight lock/statement budgets
and refuses everything else with a structured verdict (exit code 2): index
maintenance gets a pointer to the `CONCURRENTLY` idiom, and changes that need
a table rewrite — caught by the size guard or a cancelled bounded attempt —
get an explicit **not native-safe** verdict. `diff`, `fmt`, and `lint` are
still stubs. The design docs and the phased build plan live in
[docs/](docs/) — start with [docs/README.md](docs/README.md).

The codebase is partitioned into a small safety-critical core and a
periphery — **[SAFETY.md](SAFETY.md)** says which packages are which and the
Expand Down
12 changes: 7 additions & 5 deletions SAFETY.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,23 +18,25 @@ The invariant registry (invariant IDs referenced below) lives in
| Package | Core? | Status | Invariants enforced |
| --- | --- | --- | --- |
| `pkg/dbconn` — pool defaults, advisory lock, terminate-blockers, retries, RDS TLS | ✅ core | exists (Phase 0) | LK-1, LK-2 primitives |
| `pkg/preflight` — precondition verifier, refusals | ✅ core | planned (Phase 1–2) | ST-6, RF-1..RF-5 |
| `pkg/preflight` — precondition verifier, refusals | ✅ core | exists (Phase 1: table-size guard); grows through Phase 2 | ST-6, RF-1..RF-5 |
| `pkg/executor` — bounded optimistic attempt; native executor later | ✅ core | exists (Phase 1: attempt-under-budget); Executor contract at Phase 2–3 | LK-2 (attempt bound) |
| `pkg/checksum` — chunk verifier, continuous checker, repair | ✅ core | planned (Phase 5) | CO-1, CO-2, CO-3 |
| `pkg/copier` — shadow-table chunked copy | ✅ core | planned (Phase 4) | CO-4, LK-3 |
| `pkg/applier` — change apply, buffer, flush scheduling | ✅ core | planned (Phase 6) | CO-4, CO-5, CO-6, LK-3 |
| `pkg/decode` — logical decoding, LSN/position accounting | ✅ core | planned (Phase 6) | ST-4, CO-4 |
| `pkg/checkpoint` — durable resume state | ✅ core | planned (Phase 8) | ST-1, ST-2 |
| slot lifecycle (in `pkg/decode`) — create, reap, lag ceiling | ✅ core | planned (Phase 8) | ST-3 |
| `pkg/migration` — orchestrator, **cutover swap + fidelity gate** | ✅ core | planned (Phase 7) | LK-2, LK-4, ST-5 |
| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/lint` — classify/diff/route | ❌ periphery¹ | planned (Phase 1–2) | (CO-7 holds at the parse boundary) |
| `internal/cli` — CLI, flags, help, prompts | ❌ periphery | exists (stubs) | — |
| `pkg/schemachange` — orchestrator, **cutover swap + fidelity gate** | ✅ core | planned (Phase 7) | LK-2, LK-4, ST-5 |
| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/lint` — classify/diff/route | ❌ periphery¹ | `pkg/statement` exists (Phase 1: type gate); rest planned (Phase 2) | (CO-7 holds at the parse boundary) |
| `pkg/verdict` — structured outcome contract, rendering, exit codes | ❌ periphery | exists (Phase 1) | — |
| `internal/cli` — CLI, flags, help, prompts | ❌ periphery | `migrate`/`status` exist (Phase 1); rest stubs | — |
| status / progress / advisory rendering, metrics | ❌ periphery | planned | — |
| orchestrator adapter | ❌ periphery | planned (Phase 11) | OC-* hold *at* the boundary |
| `internal/testutil` | ❌ test-only | exists | — |

¹ **The planner is deliberately outside the core.** Its verdicts are *requests*, not
permissions: a wrong "native-safe" verdict is capped by the executor's own `lock_timeout` bound;
a wrong "copy" verdict produces a wasteful but *correct* migration (the checksum still gates).
a wrong "copy" verdict produces a wasteful but *correct* schema change (the checksum still gates).
The core executors re-verify their own preconditions and never trust that the planner checked.

## Rules inside the core
Expand Down
12 changes: 11 additions & 1 deletion cmd/pg-sprite/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,13 @@
package main

import (
"errors"
"os"

"github.com/alecthomas/kong"

"github.com/block/pg-sprite/internal/cli"
"github.com/block/pg-sprite/pkg/verdict"
)

// version is stamped at release time via -ldflags "-X main.version=…".
Expand All @@ -17,5 +21,11 @@ func main() {
kong.UsageOnError(),
kong.Vars{"version": version},
)
k.FatalIfErrorf(k.Run())
err := k.Run()
// A refusal verdict was already printed; its exit code is distinct from
// operational errors so automation can branch on the difference.
if errors.Is(err, verdict.ErrRefused) {
os.Exit(verdict.ExitCodeRefused)
}
k.FatalIfErrorf(err)
}
13 changes: 7 additions & 6 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,20 +48,21 @@ boundary) is defined in [../SAFETY.md](../SAFETY.md).

| Package | Role | Status |
| --- | --- | --- |
| `cmd/pg-sprite` | CLI entry point (Kong): `migrate` · `diff` · `fmt` · `lint` · `status` | exists (stubs) |
| `internal/cli` | Command tree and flag handling | exists (stubs) |
| `cmd/pg-sprite` | CLI entry point (Kong): `migrate` · `diff` · `fmt` · `lint` · `status` | `migrate`/`status` exist; rest stubs |
| `internal/cli` | Command tree and flag handling | `migrate`/`status` exist; rest stubs |
| `internal/testutil` | Test harness: containerized PostgreSQL, throwaway schemas | exists |
| `pkg/dbconn` | Pool with bounded session timeouts, retries, RDS/Aurora auto-TLS (embedded CA bundle), terminate-blockers; advisory-lock mutual exclusion lands here | exists |
| `pkg/statement` | `pg_query_go` parsing + classification (never hand-parse SQL) | Phase 1–2 |
| `pkg/preflight` | Precondition verification and refusals before any write | Phase 1–2 |
| `pkg/statement` | `go-pgquery` (Wasm `libpg_query`) parsing + classification (never hand-parse SQL); shadow DDL + fingerprints come from scratch-DB execute-and-introspect | exists (Phase 1: type gate); classification at Phase 2 |
| `pkg/preflight` | Precondition verification and refusals before any write | exists (Phase 1: table-size guard); grows through Phase 2 |
| `pkg/verdict` | Structured outcome contract (executed / refused + reason + safer idiom), rendering, exit codes | exists (Phase 1) |
| `pkg/planner` / `pkg/schemadiff` / `pkg/lint` | Shared front-end: introspect, declarative diff (may wrap [stripe/pg-schema-diff](https://github.com/stripe/pg-schema-diff) — see the low-level design's open decisions), classify, lint | Phase 2 |
| `pkg/executor` | The `Executor` contract (`Plan`/`Execute`/`Status`/`Abort`) + native executor | Phase 2–3 |
| `pkg/executor` | The `Executor` contract (`Plan`/`Execute`/`Status`/`Abort`) + native executor | exists (Phase 1: bounded optimistic attempt); contract at Phase 2–3 |
| `pkg/table` | PK-range chunkers (single-column fast path, composite), dynamic time-based sizing | Phase 4 |
| `pkg/copier` | Parallel chunked copy into the shadow table (never overwrites) | Phase 4 |
| `pkg/checksum` | The mandatory correctness gate; continuous checker; repair primitive | Phase 5 |
| `pkg/decode` | Logical-decoding change capture, LSN accounting, slot lifecycle | Phase 6, 8 |
| `pkg/applier` | Change apply onto the shadow (always wins), buffer/dedup, flush scheduling | Phase 6 |
| `pkg/migration` | Orchestrator: lifecycle, cutover swap + fidelity gate, checkpoint/resume | Phase 7–8 |
| `pkg/schemachange` | Orchestrator: lifecycle, cutover swap + fidelity gate, checkpoint/resume | Phase 7–8 |
| `pkg/checkpoint` | Durable single-row resume state | Phase 8 |
| `pkg/throttler` | Aurora reader-lag / slot-lag / WAL throttling | Phase 8 |

Expand Down
Loading