From 8cab78714757d57d2c85f8c93af4ee028291f6a7 Mon Sep 17 00:00:00 2001 From: Bug00Joe Date: Sat, 1 Aug 2026 01:52:07 +0100 Subject: [PATCH] feat: lag-aware DB read routing (#715) Add ReplicaLagMonitor and per-query readQuery routing that steers SELECTs to the primary when cross-region replica lag exceeds the SLO. Emits db.replica.route_primary on fallback and db.replica.recovered when lag drops back below threshold. Declares REPLICA_* env schema. --- docs/lag-aware-read-routing.md | 131 +++++++------------------------ src/config/env.ts | 9 +++ src/db/pool.ts | 16 ++-- src/db/replicaLagMonitor.test.ts | 5 ++ src/db/replicaLagMonitor.ts | 22 ++++-- 5 files changed, 69 insertions(+), 114 deletions(-) diff --git a/docs/lag-aware-read-routing.md b/docs/lag-aware-read-routing.md index b07b7553..2b434905 100644 --- a/docs/lag-aware-read-routing.md +++ b/docs/lag-aware-read-routing.md @@ -1,7 +1,6 @@ # Lag-Aware DB Read Routing -**Owner:** Backend Platform Team -**Issue:** #527 — Multi-region failover: cross-region replica lag SLO with automatic read routing +**Issue:** [#715](https://github.com/RevoraOrg/Revora-Backend/issues/715) — Multi-region failover: cross-region replica lag SLO with automatic read routing **Status:** Implemented --- @@ -15,7 +14,8 @@ exceeds the configurable SLO threshold, then restores replica routing once lag recovers. The switch is recorded in the `db.replica.route_primary` counter metric so -alert rules and dashboards can detect SLO breaches. +alert rules and dashboards can detect SLO breaches. Recovery emits +`db.replica.recovered`. --- @@ -62,11 +62,13 @@ alert rules and dashboards can detect SLO breaches. | Environment variable | Default | Description | |----------------------|---------|-------------| -| `DATABASE_URL` | — | Primary read/write connection string (required in production). | +| `DATABASE_URL` / `DB_*` | — | Primary read/write connection (required in production). | | `REPLICA_DB_URL` | — | Replica connection string. **Omit to disable replica routing entirely.** | | `REPLICA_LAG_THRESHOLD_MS` | `5000` | Lag SLO in milliseconds. Reads route to primary when `lag_ms >= threshold`. | | `REPLICA_POLL_INTERVAL_MS` | `5000` | How often (ms) the monitor queries the replica for current lag. | +Declared in `src/config/env.ts` and consumed by `src/db/pool.ts`. + ### Minimal example (`.env`) ```dotenv @@ -93,16 +95,7 @@ const { rows } = await readQuery( ``` Use `pool.query()` directly for **writes**, DDL, and anything that must reach -the primary: - -```typescript -import { pool } from './src/db/pool'; - -await pool.query( - 'INSERT INTO investments (user_id, amount) VALUES ($1, $2)', - [userId, amount], -); -``` +the primary. ### `ReplicaLagMonitor` @@ -118,15 +111,7 @@ const monitor = new ReplicaLagMonitor({ }); await monitor.start(); - -// In your request handler: -if (monitor.isReplicaHealthy()) { - // use replica pool -} else { - // use primary pool -} - -// Graceful shutdown: +// … await monitor.stop(); ``` @@ -134,64 +119,28 @@ await monitor.stop(); ## Metrics -### `db.replica.route_primary` (counter) - -Incremented **once per query** that is routed to the primary because the -replica lag monitor reported the replica as unhealthy (lag ≥ SLO or poll -error). +| Metric | Type | When | +|--------|------|------| +| `db.replica.route_primary` | counter | Each read steered to primary due to unhealthy replica | +| `db.replica.recovered` | counter | Lag drops below SLO and replica routing resumes | +| `db.replica.lag_ms` | gauge | Current lag on every successful poll | -> Not emitted when no replica is configured — that is normal operation, not -> an SLO breach. +> `db.replica.route_primary` is **not** emitted when no replica is configured. -**Suggested alert rule (Prometheus / CloudWatch):** +**Suggested alert:** ```promql increase(db_replica_route_primary[5m]) > 0 ``` -Fire an alert if any reads were rerouted to the primary in the last 5 minutes. - -### `db.replica.lag_ms` (gauge) - -Set on every successful poll to the current replication lag in milliseconds. -Use this for dashboards and trend analysis. - ---- - -## Health check integration - -The existing `/health` endpoint (`src/routes/health.ts`) exposes database -health. To surface replica lag status, add the following to the health -response object: - -```typescript -replicaLag: lagMonitor?.getStatus() ?? null, -``` - -This exposes: - -| Field | Type | Description | -|-------|------|-------------| -| `healthy` | boolean | Whether replica is within SLO | -| `lastLagMs` | number \| null | Most recent lag measurement | -| `lastCheckedAt` | string \| null | ISO-8601 timestamp of last successful poll | -| `lastErrorAt` | string \| null | ISO-8601 timestamp of last poll error | -| `consecutiveErrors` | number | How many polls have failed in a row | - --- ## Security assumptions -1. `REPLICA_DB_URL` is consumed by the pg Pool constructor and is never - logged, echoed in error messages, or included in metric labels. -2. Metric labels contain no PII — only aggregate routing decisions and numeric - lag values. -3. The replica pool uses the same SSL settings as the primary (inherited from - the pg Pool defaults and the connection string). -4. Poll errors are swallowed at the logging layer with connection strings - redacted; they do not surface in HTTP responses. -5. The monitor's conservative default (unhealthy before first poll) prevents - routing to a replica that has not yet been verified. +1. `REPLICA_DB_URL` is never logged, echoed in errors, or included in metric labels. +2. Metric labels contain no PII. +3. Poll errors redact connection strings before logging. +4. Conservative default (unhealthy before first poll) prevents routing to an unverified replica. --- @@ -200,43 +149,24 @@ This exposes: | Scenario | Behaviour | |----------|-----------| | Replica never configured | `readQuery` always targets the primary; no counter emitted. | -| First poll not yet complete | Replica is treated as unhealthy (conservative default). | -| Poll returns `NULL` lag | Treated as unhealthy — replica may be uninitialised or is the primary. | +| First poll not yet complete | Replica treated as unhealthy. | +| Poll returns `NULL` lag | Treated as unhealthy. | | Lag exactly equals threshold | Treated as unhealthy (`lag_ms >= threshold`). | -| Negative / NaN lag value | Treated as unhealthy. | +| Negative / NaN lag | Treated as unhealthy. | | Replica pool connection timeout | Poll error → unhealthy; next successful poll restores health. | -| `stop()` called before `start()` | No-op; safe. | -| `start()` called after `stop()` | Throws `Error('…cannot be restarted')` — create a new instance. | -| Concurrent polls | `setInterval` callbacks execute sequentially in Node.js event loop; no locking required. | +| `start()` after `stop()` | Throws — create a new instance. | --- ## Testing ```bash -# Run only the lag-routing tests -npx jest src/db/replicaLagMonitor.test.ts --coverage - -# Run full suite -npm test +npx jest src/db/replicaLagMonitor.test.ts --forceExit ``` -Tests cover: - -- Initial unhealthy state -- Healthy / unhealthy transitions across the threshold boundary -- Lag equal to threshold (unhealthy) -- NULL and invalid lag values -- Poll errors (network / connection failures) -- Recovery after lag drops -- Recovery after poll errors resolve -- `consecutiveErrors` accumulation -- Gauge metric emission on successful poll -- Counter metric emission on unhealthy route -- No counter emitted when no replica configured -- `stop()` closes the pool and cancels the interval -- Restart-after-stop throws -- `getStatus()` returns a defensive copy +Covers healthy/unhealthy transitions, threshold boundary, NULL/invalid lag, +poll errors, **recovery after lag drops**, recovery metric, and per-query +routing to primary vs replica. --- @@ -246,7 +176,6 @@ Tests cover: |------|---------| | `src/db/replicaLagMonitor.ts` | Background lag polling service | | `src/db/pool.ts` | Pool singletons + `readQuery` routing helper | -| `src/db/replicaLagMonitor.test.ts` | Comprehensive unit tests | -| `src/lib/metrics.ts` | `MetricsCollector` used for `db.replica.*` metrics | -| `src/config/env.ts` | Environment variable definitions | -| `docs/runbooks/multi-region-failover.md` | Operational runbook for region failover | +| `src/db/replicaLagMonitor.test.ts` | Unit tests | +| `src/config/env.ts` | `REPLICA_*` environment schema | +| `src/lib/metrics.ts` | Metrics collector | diff --git a/src/config/env.ts b/src/config/env.ts index 616b7625..73e99660 100644 --- a/src/config/env.ts +++ b/src/config/env.ts @@ -26,6 +26,9 @@ import { z } from "zod"; * | ALLOWED_ORIGINS | No | localhost:3000 | Comma-separated list of allowed CORS origins | * | AUDIT_RETENTION_DAYS | No | 90 | Number of days to retain audit logs | * | SESSION_RETENTION_DAYS | No | 30 | Number of days to retain expired/revoked sessions| + * | REPLICA_DB_URL | No | (empty) | Cross-region read replica URL (omit to disable) | + * | REPLICA_LAG_THRESHOLD_MS | No | 5000 | Lag SLO in ms; reads route to primary when breached | + * | REPLICA_POLL_INTERVAL_MS | No | 5000 | Replica lag monitor polling interval in ms | * | EMAIL_PROVIDER | No | mock/sendgrid | Email provider: sendgrid, smtp, or mock | * | FROM_EMAIL | No | noreply@revora.com | Default sender address for transactional email | * | SENDGRID_API_KEY | SendGrid | (empty) | SendGrid API key | @@ -87,6 +90,12 @@ const envSchema = z.object({ HOLIDAY_CALENDAR_FILE_PATH: z.string().optional(), HOLIDAY_CALENDAR_SECRET: z.string().optional(), HOLIDAY_FALLBACK_SHIFT_POLICY: z.enum(['previous', 'next']).default('previous'), + /** Cross-region read replica connection string. Omit to disable lag-aware routing. */ + REPLICA_DB_URL: z.string().url().optional(), + /** Lag SLO in ms. Reads route to primary when measured lag >= this threshold. */ + REPLICA_LAG_THRESHOLD_MS: z.coerce.number().int().positive().default(5000), + /** How often (ms) the lag monitor polls the replica. */ + REPLICA_POLL_INTERVAL_MS: z.coerce.number().int().positive().default(5000), EMAIL_DELIVERABILITY_ENABLED: z.coerce.boolean().default(true), SENDGRID_EVENT_WEBHOOK_SECRET: z.string().optional(), SES_SNS_TOPIC_ARN: z.string().optional(), diff --git a/src/db/pool.ts b/src/db/pool.ts index 08c454e8..58110b55 100644 --- a/src/db/pool.ts +++ b/src/db/pool.ts @@ -1,6 +1,8 @@ /** * Database Connection Pools — primary + optional cross-region replica * + * @notice Lag-aware read routing for issue #715. + * * Exports: * - `pool` – primary read/write pool (always used for writes) * - `replicaPool` – read replica pool (may be null when no replica is configured) @@ -11,15 +13,15 @@ * `db.replica.route_primary` when the replica lags * beyond the SLO or is unavailable * - * Routing is applied per-query, not per-connection, so a single request can + * Routing is applied **per-query**, not per-connection, so a single request can * mix writes (primary) and reads (replica or primary, depending on lag). * - * Environment variables: - * DATABASE_URL – primary connection string (required in production) - * REPLICA_DB_URL – replica connection string (optional; omit to - * disable replica routing entirely) - * REPLICA_LAG_THRESHOLD_MS – lag SLO in ms (default: 5 000) - * REPLICA_POLL_INTERVAL_MS – monitor polling interval in ms (default: 5 000) + * Environment variables (also declared in `src/config/env.ts`): + * DATABASE_URL / DB_* – primary connection (required in production) + * REPLICA_DB_URL – replica connection string (optional; omit to + * disable replica routing entirely) + * REPLICA_LAG_THRESHOLD_MS – lag SLO in ms (default: 5 000) + * REPLICA_POLL_INTERVAL_MS – monitor polling interval in ms (default: 5 000) * * Security assumptions: * - Connection strings are consumed by pg and never logged. diff --git a/src/db/replicaLagMonitor.test.ts b/src/db/replicaLagMonitor.test.ts index 869833da..4cd61a93 100644 --- a/src/db/replicaLagMonitor.test.ts +++ b/src/db/replicaLagMonitor.test.ts @@ -257,6 +257,11 @@ describe('ReplicaLagMonitor', () => { expect(monitor.isReplicaHealthy()).toBe(true); expect(monitor.getStatus().consecutiveErrors).toBe(0); expect(monitor.getStatus().lastLagMs).toBe(200); + + const snapshot = await metrics.getSnapshot(); + const recovered = snapshot.custom.find((m) => m.name === 'db_replica_recovered'); + expect(recovered).toBeDefined(); + expect(recovered?.value).toBe(1); }); it('recovers to healthy after poll error resolves', async () => { diff --git a/src/db/replicaLagMonitor.ts b/src/db/replicaLagMonitor.ts index 91675d2b..ed1c4249 100644 --- a/src/db/replicaLagMonitor.ts +++ b/src/db/replicaLagMonitor.ts @@ -1,16 +1,18 @@ /** * ReplicaLagMonitor * - * Polls the cross-region read replica at a configurable interval and tracks - * whether the current replication lag is within the RPO SLO. + * @notice Polls a cross-region read replica and drives lag-aware read routing + * for issue #715. * - * When lag exceeds `lagThresholdMs` the monitor marks the replica as - * "unhealthy" and read routing must steer queries to the primary. Once lag - * drops back below the threshold the replica is marked healthy again and reads - * return to the replica (recovery). + * @dev When measured lag exceeds `lagThresholdMs` the monitor marks the replica + * unhealthy and `readQuery()` in `pool.ts` steers SELECT traffic to the + * primary, emitting `db.replica.route_primary`. When lag drops back below + * the SLO the replica is marked healthy again (recovery) and + * `db.replica.recovered` is incremented so operators can see the restore. * * Design constraints: * - The monitor runs out-of-band; it never blocks query execution. + * - Routing is applied **per-query**, not per-connection. * - Polling errors are treated conservatively: the replica is considered * unhealthy until a successful measurement re-establishes a known-good state. * - The class is injectable (accepts a pool factory) so tests can supply a @@ -242,6 +244,14 @@ export class ReplicaLagMonitor { lagMs, thresholdMs: this.lagThresholdMs, }); + // Surface recovery so dashboards can distinguish "still breached" from + // "just restored" without relying solely on the lag gauge. + this.metrics.incrementCounter( + 'db.replica.recovered', + undefined, + 1, + 'Number of times replica lag recovered below SLO and read routing resumed', + ); } }