Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Release

on:
push:
tags:
- "v*"

permissions:
contents: write

jobs:
release:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
persist-credentials: false

- name: Set up Go
uses: actions/setup-go@v5
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
with:
go-version-file: go.mod
# Release artifacts must never be built from a shared cache a
# previous workflow could have poisoned.
cache: false

- name: Run GoReleaser
uses: goreleaser/goreleaser-action@9c156ee8a17a598857849441385a2041ef570552 # v6.3.0
with:
version: v2.8.2
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
45 changes: 45 additions & 0 deletions .goreleaser.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
version: 2

builds:
- id: pg-sprite
main: ./cmd/pg-sprite
binary: pg-sprite
# The SQL parser is Wasm (wasilibs/go-pgquery), so the binary is pure Go
# and cross-compiles without cgo.
env:
- CGO_ENABLED=0
ldflags:
- -X main.version={{.Tag}}
goos:
- linux
- darwin
goarch:
- amd64
- arm64
ignore:
- goos: darwin
goarch: amd64

archives:
- id: pg-sprite
builds:
- pg-sprite
format: tar.gz
name_template: "pg-sprite_{{ .Version }}_{{ .Os }}_{{ .Arch }}"

release:
# If a release already exists for the tag, replace its assets rather than
# failing. This handles the case where a release is created via the GitHub
# UI before the workflow runs.
mode: replace

checksum:
name_template: "checksums.txt"

changelog:
sort: asc
filters:
exclude:
- "^docs:"
- "^test:"
- "^ci:"
28 changes: 18 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,17 +14,25 @@ PostgreSQL 14+): a decoupled **planner → router → executor** design where th
planner classifies each change, the router picks a strategy, and
interchangeable executors carry it out — the cheap native PostgreSQL idiom
when one exists (`CONCURRENTLY`, `NOT VALID` + `VALIDATE`, fast default,
`USING INDEX`), and a log-based, checksum-gated, resumable copy-and-swap when
a genuine table rewrite is unavoidable.
`USING INDEX`), while a log-based, checksum-gated, resumable copy-and-swap for
genuine table rewrites lands in a later phase.

**Status: Phase 1 (optimistic front door).** `pg-sprite migrate --alter '…'`
runs easy `ALTER TABLE` changes directly under tight lock/statement budgets
and refuses everything else with a structured verdict (exit code 2): index
maintenance gets a pointer to the `CONCURRENTLY` idiom, and changes that need
a table rewrite — caught by the size guard or a cancelled bounded attempt —
get an explicit **not native-safe** verdict. `diff`, `fmt`, and `lint` are
still stubs. The design docs and the phased build plan live in
[docs/](docs/) — start with [docs/README.md](docs/README.md).
The planner is PostgreSQL's missing `ALGORITHM=` / `LOCK=` declaration: MySQL
lets authors assert a cost bracket and a concurrency impact and fails closed
when either can't be honored — PostgreSQL silently runs whichever cost
applies. pg-sprite proves both dimensions before execution, routes each
change to the safest sequence that exists, and refuses with a structured
verdict when it can't prove one (see
[docs/postgres-online-ddl-reference.md](docs/postgres-online-ddl-reference.md)).

**Status: Phases 1 and 2.1–2.4.** The parse boundary, declarative diff,
classifier, and router seam are implemented. `pg-sprite migrate --alter '…'`
runs a bounded optimistic native attempt; routed execution beyond that attempt
lands in Phase 3. Changes without an available backend get a structured
refusal (exit code 2). `lint` is still a stub. The design docs and the phased
build plan live in [docs/](docs/) — start with
[docs/README.md](docs/README.md); the vision — what pg-sprite is and is not —
is [docs/vision.md](docs/vision.md).

The codebase is partitioned into a small safety-critical core and a
periphery — **[SAFETY.md](SAFETY.md)** says which packages are which and the
Expand Down
23 changes: 12 additions & 11 deletions SAFETY.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The invariant registry (invariant IDs referenced below) lives in

| Package | Core? | Status | Invariants enforced |
| --- | --- | --- | --- |
| `pkg/dbconn` — pool defaults, advisory lock, terminate-blockers, retries, RDS TLS | ✅ core | exists (Phase 0) | LK-1, LK-2 primitives |
| `pkg/dbconn` — pool defaults, terminate-blockers, retries, RDS TLS; advisory table lock planned | ✅ core | exists; advisory table lock planned | LK-2 primitives; LK-1 planned |
| `pkg/preflight` — precondition verifier, refusals | ✅ core | exists (Phase 1: table-size guard); grows through Phase 2 | ST-6, RF-1..RF-5 |
| `pkg/executor` — bounded optimistic attempt; native executor later | ✅ core | exists (Phase 1: attempt-under-budget); Executor contract at Phase 2–3 | LK-2 (attempt bound) |
| `pkg/checksum` — chunk verifier, continuous checker, repair | ✅ core | planned (Phase 5) | CO-1, CO-2, CO-3 |
Expand All @@ -27,16 +27,17 @@ The invariant registry (invariant IDs referenced below) lives in
| `pkg/checkpoint` — durable resume state | ✅ core | planned (Phase 8) | ST-1, ST-2 |
| slot lifecycle (in `pkg/decode`) — create, reap, lag ceiling | ✅ core | planned (Phase 8) | ST-3 |
| `pkg/migration` — orchestrator, **cutover swap + fidelity gate** | ✅ core | planned (Phase 7) | LK-2, LK-4, ST-5 |
| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/router`, `pkg/lint` — classify/diff/route | ❌ periphery¹ | `pkg/statement` (parse boundary), `pkg/schemadiff` (introspect/diff via scratch execute-and-introspect), `pkg/planner` (classifier), and `pkg/router` (backend assignment + availability policy) exist (Phase 2); `pkg/lint` planned (Phase 2) | (CO-7 holds at the parse boundary) |
| `pkg/statement`, `pkg/planner`, `pkg/schemadiff`, `pkg/router`, `pkg/lint` — classify/diff/route | ❌ periphery¹ | `pkg/statement` (parse boundary), `pkg/schemadiff` (introspect/diff via scratch execute-and-introspect), `pkg/planner` (classifier), and `pkg/router` (backend assignment + availability policy) exist (Phases 2.1–2.4); `pkg/lint` planned | (CO-7 holds at the parse boundary) |
| `pkg/verdict` — structured outcome contract, rendering, exit codes | ❌ periphery | exists (Phase 1) | — |
| `internal/cli` — CLI, flags, help, prompts | ❌ periphery | `migrate`/`status` exist (Phase 1); rest stubs | — |
| `internal/cli` — CLI, flags, help, prompts | ❌ periphery | `migrate`, `status`, `diff`, and `fmt` exist; `lint` is a stub | — |
| status / progress / advisory rendering, metrics | ❌ periphery | planned | — |
| orchestrator adapter | ❌ periphery | planned (Phase 11) | OC-* hold *at* the boundary |
| `internal/testutil` | ❌ test-only | exists | — |

¹ **The planner is deliberately outside the core.** Its verdicts are *requests*, not
permissions: a wrong "native-safe" verdict is capped by the executor's own `lock_timeout` bound;
a wrong "copy" verdict produces a wasteful but *correct* migration (the checksum still gates).
permissions: a wrong "native-safe" verdict is capped by the executor's own `lock_timeout` bound.
Today a "copy" route reports unavailable; once copy-and-swap exists, a wrong "copy" verdict will
produce a wasteful but *correct* migration because the checksum will still gate it.
The core executors re-verify their own preconditions and never trust that the planner checked.

## Rules inside the core
Expand All @@ -46,18 +47,18 @@ The short version — the full rules live in [docs/tcb-model.md](docs/tcb-model.
- **Never trust callers.** Every dangerous operation re-verifies its preconditions, whoever the
requester is (CLI, planner, orchestrator). The periphery may request; the core enforces.
- **Domain types make illegal states unrepresentable.** Validating passages return proof types
with package-private constructors (`statement.Classified`, `PreflightedTable`,
`VerifiedShadow`, `CleanWatermark`, `TableLock`); dangerous APIs accept only proof types —
e.g. the cutover swap accepts only a `VerifiedShadow`.
with package-private constructors (today `preflight.PreflightedTable`; later phases add
`VerifiedShadow`, `CleanWatermark`, and `TableLock`); dangerous APIs accept only proof types —
e.g. the planned cutover swap will accept only a `VerifiedShadow`.
- **Put a limit on everything.** Every loop bounded, every queue bounded, every retry counted,
every wait deadlined. An unbounded anything in a core package is a review-blocking defect.
- **Assert the positive and the negative space; pair assertions across boundaries.** Invariant
violations use a distinct error class (`ErrInvariantViolation`) naming the invariant ID, and
always abort fail-closed — never a warning, never retried.
violations will use a distinct error class (`ErrInvariantViolation`) naming the invariant ID
once the executor phases land, and always abort fail-closed — never a warning, never retried.
- **Locality of behavior.** The enforcement point of an invariant carries a `// INV: <id>`
comment so a reviewer or agent can grep the ID and see the whole enforcement in one screen.
- **Dependencies inside the core become part of the core.** Current core dependency list:
`pgx/v5`, `pglogrepl`, stdlib. Adding one requires a recorded decision (see the rubric in
`pgx/v5`, stdlib. The future decode path will add `pglogrepl`. Adding one requires a recorded decision (see the rubric in
[docs/tcb-model.md](docs/tcb-model.md) — copy small things, take pinned dependencies only
for load-bearing expertise).
pg-sprite **never imports `block/spirit` as a module**: we port ideas with citations, not
Expand Down
22 changes: 11 additions & 11 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,17 @@ checkpoint/resume, tuned for Aurora. That is the gap this engine targets.

| Doc | Contents |
| --- | --- |
| [vision.md](vision.md) | The **vision statement** — what pg-sprite is (the reliable execution engine under a GitOps front-end like [SchemaBot](https://github.com/block/schemabot), as [Spirit](https://github.com/block/spirit) is for MySQL) and what it deliberately is not. Five pillars, success criteria, and explicit non-goals. Start here for the why. |
| [architecture.md](architecture.md) | The **one-screen codebase map** — the three layers, the package map with build status, the copy-and-swap lifecycle, and where to read more. Start here for orientation. |
| [postgres-online-ddl-reference.md](postgres-online-ddl-reference.md) | The Aurora PostgreSQL equivalent of MySQL's [InnoDB Online DDL Operations](https://dev.mysql.com/doc/refman/8.4/en/innodb-online-ddl-operations.html) reference — lock levels, rewrite/scan behaviour, and concurrent-DML safety per operation. |
| [high-level-design.md](high-level-design.md) | The **high-level design** — the conceptual overview: the problem, the planner → router → executor philosophy, the execution patterns and when each is chosen, and coverage at a glance. No package/interface detail. Start here for the architecture. |
| [low-level-design.md](low-level-design.md) | The **low-level design** — the detailed engineering design: package layout, the `Executor` interface, library choices, copy-and-swap lifecycle internals, the full coverage matrix, table requirements, and open decisions. Read this when designing the interfaces and packages. |
| [low-level-design.md](low-level-design.md) | The **low-level design** — the detailed engineering design: package layout, the `Executor` interface, library choices, copy-and-swap lifecycle internals, the full coverage matrix, table requirements, and the decisions remaining for later execution phases. Read this when designing the interfaces and packages. |
| [design-principles.md](design-principles.md) | The canonical **design principles** that govern the engine — safety over speed, decisions-not-options, classify-first, mandatory checksum gate, log-based CDC, and the PostgreSQL/Aurora-specific rules everything else traces back to. |
| [postgresql-version-support.md](postgresql-version-support.md) | The **PostgreSQL version matrix** — which PG majors pgroll, pg_osc, and pg_repack support, which majors Aurora still ships, the minimum PG version each native idiom needs, and the resulting decision to **pivot on PostgreSQL 14+** (validated 14 → 18). |
| [change-capture-tradeoff.md](change-capture-tradeoff.md) | The canonical **triggers vs logical-decoding** trade-off for copy-and-swap — overhead, failover survival, WAL risk, and whether either lets us drop the checksum/checkpoint (answer: keep the checksum; triggers simplify but don't remove the checkpoint). Any doc proposing logical decoding as the default points here. |
| [invariants.md](invariants.md) | The canonical **invariant registry** — testable runtime MUST-statements (correctness, locking, state/resume, refusals, orchestration), each with its enforcement point and source. Mined from this doc set plus [Spirit](https://github.com/block/spirit)'s stated safety invariants and [SchemaBot](https://github.com/block/schemabot)'s control-plane discipline; the build plan's phases carry per-invariant test obligations. |
| [tcb-model.md](tcb-model.md) | The **TCB model** — the trusted-computing-base partition of the engine: which components are the small trusted core that enforces the invariant registry vs the untrusted periphery, the never-trust-callers rule, domain types that make illegal states unrepresentable, the in-TCB engineering rules (from TigerBeetle TIGER_STYLE, s2n-tls, qmail, bitcoin-core), the verification ladder, and the per-side AI-assisted development policy. |
| [testing.md](testing.md) | The **test-suite guide** — how to run the suite (unit, per-major, all supported majors, compose database), what Phase 0 covers today, the per-phase deferred test obligations, and the vanilla-PostgreSQL-matrix vs real-Aurora validation boundary. |
| [testing.md](testing.md) | The **test-suite guide** — how to run the suite (unit, per-major, all supported majors, compose database), current coverage, the remaining executor-phase test obligations, and the vanilla-PostgreSQL-matrix vs real-Aurora validation boundary. |
| [schemabot-integration.md](schemabot-integration.md) | The **single home for orchestrator integration** — how SchemaBot (the reference orchestrator) drives the engine: the pluggable-engine overview, the verb mappings, the concrete adapter contract, and the design constraints (OC-* invariants) the integration imposes on the core. |

## TL;DR recommendation
Expand All @@ -67,10 +68,10 @@ chunking*, and *checkpoint/resume*.
simply *attempts* the change under a tight `lock_timeout` + `statement_timeout`; if it
completes it was effectively instant/in-place, and if it can't it is cancelled and treated as
a rewrite. This is the analog of Spirit's "attempt INSTANT/INPLACE first" — adapted to a
database with no instant-or-error assertion. **Full classification** (parse-based) comes next
and *predicts* the path up front (`CREATE INDEX CONCURRENTLY`, `ADD ... NOT VALID` +
`VALIDATE`, PG11+ fast default, `ADD PK USING INDEX`, binary-coercible type change), powering
dry-run, advisory, and the declarative diff.
database with no instant-or-error assertion. **Full classification** (parse-based) is
implemented in `pkg/planner` and *predicts* the path up front (`CREATE INDEX CONCURRENTLY`,
`ADD ... NOT VALID` + `VALIDATE`, PG11+ fast default, `ADD PK USING INDEX`, binary-coercible
type change), powering dry-run, advisory, and the declarative diff.
2. **Otherwise copy — refuse honestly until the engine exists.** For genuine table rewrites
(`ALTER COLUMN TYPE` general, volatile-default `ADD COLUMN`, `STORED` generated column,
repack), the **near-term** stance is a clear **refusal with the classification and reason** —
Expand All @@ -85,13 +86,12 @@ chunking*, and *checkpoint/resume*.
environments that cannot enable `rds.logical_replication` or can't accept slot loss on
failover. This default is cluster-dependent, not absolute — see the
[change-capture trade-off](change-capture-tradeoff.md).
4. **Two front-ends, one pipeline — build declarative first.** *Declarative* (`diff`/`fmt`)
4. **Two front-ends, one pipeline.** *Declarative* (`diff`/`fmt`)
lets the user submit a desired `CREATE TABLE` and derives the `ALTER` by diffing against the
live schema (the analog of Spirit's declarative workflow). Build this first; the *imperative*
(`--alter`) path is then a trivial add-on — the **same** classify → native-or-copy pipeline
with the diff step skipped (the user's `ALTER` goes straight into the classifier).
live schema (the analog of Spirit's declarative workflow). It and the *imperative*
(`--alter`) path both exist and share the **same** classify → route pipeline; declarative
mode adds the diff step, while imperative mode skips it.

See [high-level-design.md](high-level-design.md) for the conceptual architecture, then
[low-level-design.md](low-level-design.md) for the package/interface detail and the open
decisions.

Loading
Loading