feat: lag-aware DB read routing (#715)

docs/lag-aware-read-routing.md

@@ -1,7 +1,6 @@
 # Lag-Aware DB Read Routing
 
-**Owner:** Backend Platform Team  
-**Issue:** #527 — Multi-region failover: cross-region replica lag SLO with automatic read routing  
+**Issue:** [#715](https://github.com/RevoraOrg/Revora-Backend/issues/715) — Multi-region failover: cross-region replica lag SLO with automatic read routing  
 **Status:** Implemented
 
 ---
@@ -15,7 +14,8 @@ exceeds the configurable SLO threshold, then restores replica routing once lag
 recovers.
 
 The switch is recorded in the `db.replica.route_primary` counter metric so
-alert rules and dashboards can detect SLO breaches.
+alert rules and dashboards can detect SLO breaches. Recovery emits
+`db.replica.recovered`.
 
 ---
 
@@ -62,11 +62,13 @@ alert rules and dashboards can detect SLO breaches.
 
 | Environment variable | Default | Description |
 |----------------------|---------|-------------|
-| `DATABASE_URL` | — | Primary read/write connection string (required in production). |
+| `DATABASE_URL` / `DB_*` | — | Primary read/write connection (required in production). |
 | `REPLICA_DB_URL` | — | Replica connection string. **Omit to disable replica routing entirely.** |
 | `REPLICA_LAG_THRESHOLD_MS` | `5000` | Lag SLO in milliseconds. Reads route to primary when `lag_ms >= threshold`. |
 | `REPLICA_POLL_INTERVAL_MS` | `5000` | How often (ms) the monitor queries the replica for current lag. |
 
+Declared in `src/config/env.ts` and consumed by `src/db/pool.ts`.
+
 ### Minimal example (`.env`)
 
 ```dotenv
@@ -93,16 +95,7 @@ const { rows } = await readQuery<User>(
 ```
 
 Use `pool.query()` directly for **writes**, DDL, and anything that must reach
-the primary:
-
-```typescript
-import { pool } from './src/db/pool';
-
-await pool.query(
-  'INSERT INTO investments (user_id, amount) VALUES ($1, $2)',
-  [userId, amount],
-);
-```
+the primary.
 
 ### `ReplicaLagMonitor`
 
@@ -118,80 +111,36 @@ const monitor = new ReplicaLagMonitor({
 });
 
 await monitor.start();
-
-// In your request handler:
-if (monitor.isReplicaHealthy()) {
-  // use replica pool
-} else {
-  // use primary pool
-}
-
-// Graceful shutdown:
+// …
 await monitor.stop();
 ```
 
 ---
 
 ## Metrics
 
-### `db.replica.route_primary` (counter)
-
-Incremented **once per query** that is routed to the primary because the
-replica lag monitor reported the replica as unhealthy (lag ≥ SLO or poll
-error).
+| Metric | Type | When |
+|--------|------|------|
+| `db.replica.route_primary` | counter | Each read steered to primary due to unhealthy replica |
+| `db.replica.recovered` | counter | Lag drops below SLO and replica routing resumes |
+| `db.replica.lag_ms` | gauge | Current lag on every successful poll |
 
-> Not emitted when no replica is configured — that is normal operation, not
-> an SLO breach.
+> `db.replica.route_primary` is **not** emitted when no replica is configured.
 
-**Suggested alert rule (Prometheus / CloudWatch):**
+**Suggested alert:**
 
 ```promql
 increase(db_replica_route_primary[5m]) > 0
 ```
 
-Fire an alert if any reads were rerouted to the primary in the last 5 minutes.
-
-### `db.replica.lag_ms` (gauge)
-
-Set on every successful poll to the current replication lag in milliseconds.
-Use this for dashboards and trend analysis.
-
----
-
-## Health check integration
-
-The existing `/health` endpoint (`src/routes/health.ts`) exposes database
-health. To surface replica lag status, add the following to the health
-response object:
-
-```typescript
-replicaLag: lagMonitor?.getStatus() ?? null,
-```
-
-This exposes:
-
-| Field | Type | Description |
-|-------|------|-------------|
-| `healthy` | boolean | Whether replica is within SLO |
-| `lastLagMs` | number \| null | Most recent lag measurement |
-| `lastCheckedAt` | string \| null | ISO-8601 timestamp of last successful poll |
-| `lastErrorAt` | string \| null | ISO-8601 timestamp of last poll error |
-| `consecutiveErrors` | number | How many polls have failed in a row |
-
 ---
 
 ## Security assumptions
 
-1. `REPLICA_DB_URL` is consumed by the pg Pool constructor and is never
-   logged, echoed in error messages, or included in metric labels.
-2. Metric labels contain no PII — only aggregate routing decisions and numeric
-   lag values.
-3. The replica pool uses the same SSL settings as the primary (inherited from
-   the pg Pool defaults and the connection string).
-4. Poll errors are swallowed at the logging layer with connection strings
-   redacted; they do not surface in HTTP responses.
-5. The monitor's conservative default (unhealthy before first poll) prevents
-   routing to a replica that has not yet been verified.
+1. `REPLICA_DB_URL` is never logged, echoed in errors, or included in metric labels.
+2. Metric labels contain no PII.
+3. Poll errors redact connection strings before logging.
+4. Conservative default (unhealthy before first poll) prevents routing to an unverified replica.
 
 ---
 
@@ -200,43 +149,24 @@ This exposes:
 | Scenario | Behaviour |
 |----------|-----------|
 | Replica never configured | `readQuery` always targets the primary; no counter emitted. |
-| First poll not yet complete | Replica is treated as unhealthy (conservative default). |
-| Poll returns `NULL` lag | Treated as unhealthy — replica may be uninitialised or is the primary. |
+| First poll not yet complete | Replica treated as unhealthy. |
+| Poll returns `NULL` lag | Treated as unhealthy. |
 | Lag exactly equals threshold | Treated as unhealthy (`lag_ms >= threshold`). |
-| Negative / NaN lag value | Treated as unhealthy. |
+| Negative / NaN lag | Treated as unhealthy. |
 | Replica pool connection timeout | Poll error → unhealthy; next successful poll restores health. |
-| `stop()` called before `start()` | No-op; safe. |
-| `start()` called after `stop()` | Throws `Error('…cannot be restarted')` — create a new instance. |
-| Concurrent polls | `setInterval` callbacks execute sequentially in Node.js event loop; no locking required. |
+| `start()` after `stop()` | Throws — create a new instance. |
 
 ---
 
 ## Testing
 
 ```bash
-# Run only the lag-routing tests
-npx jest src/db/replicaLagMonitor.test.ts --coverage
-
-# Run full suite
-npm test
+npx jest src/db/replicaLagMonitor.test.ts --forceExit
 ```
 
-Tests cover:
-
-- Initial unhealthy state
-- Healthy / unhealthy transitions across the threshold boundary
-- Lag equal to threshold (unhealthy)
-- NULL and invalid lag values
-- Poll errors (network / connection failures)
-- Recovery after lag drops
-- Recovery after poll errors resolve
-- `consecutiveErrors` accumulation
-- Gauge metric emission on successful poll
-- Counter metric emission on unhealthy route
-- No counter emitted when no replica configured
-- `stop()` closes the pool and cancels the interval
-- Restart-after-stop throws
-- `getStatus()` returns a defensive copy
+Covers healthy/unhealthy transitions, threshold boundary, NULL/invalid lag,
+poll errors, **recovery after lag drops**, recovery metric, and per-query
+routing to primary vs replica.
 
 ---
 
@@ -246,7 +176,6 @@ Tests cover:
 |------|---------|
 | `src/db/replicaLagMonitor.ts` | Background lag polling service |
 | `src/db/pool.ts` | Pool singletons + `readQuery` routing helper |
-| `src/db/replicaLagMonitor.test.ts` | Comprehensive unit tests |
-| `src/lib/metrics.ts` | `MetricsCollector` used for `db.replica.*` metrics |
-| `src/config/env.ts` | Environment variable definitions |
-| `docs/runbooks/multi-region-failover.md` | Operational runbook for region failover |
+| `src/db/replicaLagMonitor.test.ts` | Unit tests |
+| `src/config/env.ts` | `REPLICA_*` environment schema |
+| `src/lib/metrics.ts` | Metrics collector |

src/config/env.ts

@@ -26,6 +26,9 @@ import { z } from "zod";
  * | ALLOWED_ORIGINS             | No       | localhost:3000          | Comma-separated list of allowed CORS origins     |
  * | AUDIT_RETENTION_DAYS        | No       | 90                      | Number of days to retain audit logs              |
  * | SESSION_RETENTION_DAYS      | No       | 30                      | Number of days to retain expired/revoked sessions|
+ * | REPLICA_DB_URL              | No       | (empty)                 | Cross-region read replica URL (omit to disable)  |
+ * | REPLICA_LAG_THRESHOLD_MS    | No       | 5000                    | Lag SLO in ms; reads route to primary when breached |
+ * | REPLICA_POLL_INTERVAL_MS    | No       | 5000                    | Replica lag monitor polling interval in ms       |
  * | EMAIL_PROVIDER              | No       | mock/sendgrid           | Email provider: sendgrid, smtp, or mock          |
  * | FROM_EMAIL                  | No       | noreply@revora.com      | Default sender address for transactional email   |
  * | SENDGRID_API_KEY            | SendGrid | (empty)                 | SendGrid API key                                 |
@@ -87,6 +90,12 @@ const envSchema = z.object({
   HOLIDAY_CALENDAR_FILE_PATH: z.string().optional(),
   HOLIDAY_CALENDAR_SECRET: z.string().optional(),
   HOLIDAY_FALLBACK_SHIFT_POLICY: z.enum(['previous', 'next']).default('previous'),
+  /** Cross-region read replica connection string. Omit to disable lag-aware routing. */
+  REPLICA_DB_URL: z.string().url().optional(),
+  /** Lag SLO in ms. Reads route to primary when measured lag >= this threshold. */
+  REPLICA_LAG_THRESHOLD_MS: z.coerce.number().int().positive().default(5000),
+  /** How often (ms) the lag monitor polls the replica. */
+  REPLICA_POLL_INTERVAL_MS: z.coerce.number().int().positive().default(5000),
   EMAIL_DELIVERABILITY_ENABLED: z.coerce.boolean().default(true),
   SENDGRID_EVENT_WEBHOOK_SECRET: z.string().optional(),
   SES_SNS_TOPIC_ARN: z.string().optional(),

src/db/pool.ts

@@ -1,6 +1,8 @@
 /**
  * Database Connection Pools — primary + optional cross-region replica
  *
+ * @notice Lag-aware read routing for issue #715.
+ *
  * Exports:
  *   - `pool`        – primary read/write pool (always used for writes)
  *   - `replicaPool` – read replica pool (may be null when no replica is configured)
@@ -11,15 +13,15 @@
  *                         `db.replica.route_primary` when the replica lags
  *                         beyond the SLO or is unavailable
  *
- * Routing is applied per-query, not per-connection, so a single request can
+ * Routing is applied **per-query**, not per-connection, so a single request can
  * mix writes (primary) and reads (replica or primary, depending on lag).
  *
- * Environment variables:
- *   DATABASE_URL            – primary connection string (required in production)
- *   REPLICA_DB_URL          – replica connection string (optional; omit to
- *                             disable replica routing entirely)
- *   REPLICA_LAG_THRESHOLD_MS – lag SLO in ms (default: 5 000)
- *   REPLICA_POLL_INTERVAL_MS – monitor polling interval in ms (default: 5 000)
+ * Environment variables (also declared in `src/config/env.ts`):
+ *   DATABASE_URL / DB_*       – primary connection (required in production)
+ *   REPLICA_DB_URL            – replica connection string (optional; omit to
+ *                               disable replica routing entirely)
+ *   REPLICA_LAG_THRESHOLD_MS  – lag SLO in ms (default: 5 000)
+ *   REPLICA_POLL_INTERVAL_MS  – monitor polling interval in ms (default: 5 000)
  *
  * Security assumptions:
  *   - Connection strings are consumed by pg and never logged.

src/db/replicaLagMonitor.test.ts

@@ -257,6 +257,11 @@ describe('ReplicaLagMonitor', () => {
     expect(monitor.isReplicaHealthy()).toBe(true);
     expect(monitor.getStatus().consecutiveErrors).toBe(0);
     expect(monitor.getStatus().lastLagMs).toBe(200);
+
+    const snapshot = await metrics.getSnapshot();
+    const recovered = snapshot.custom.find((m) => m.name === 'db_replica_recovered');
+    expect(recovered).toBeDefined();
+    expect(recovered?.value).toBe(1);
   });
 
   it('recovers to healthy after poll error resolves', async () => {

src/db/replicaLagMonitor.ts

@@ -1,16 +1,18 @@
 /**
  * ReplicaLagMonitor
  *
- * Polls the cross-region read replica at a configurable interval and tracks
- * whether the current replication lag is within the RPO SLO.
+ * @notice Polls a cross-region read replica and drives lag-aware read routing
+ *         for issue #715.
  *
- * When lag exceeds `lagThresholdMs` the monitor marks the replica as
- * "unhealthy" and read routing must steer queries to the primary.  Once lag
- * drops back below the threshold the replica is marked healthy again and reads
- * return to the replica (recovery).
+ * @dev When measured lag exceeds `lagThresholdMs` the monitor marks the replica
+ *      unhealthy and `readQuery()` in `pool.ts` steers SELECT traffic to the
+ *      primary, emitting `db.replica.route_primary`.  When lag drops back below
+ *      the SLO the replica is marked healthy again (recovery) and
+ *      `db.replica.recovered` is incremented so operators can see the restore.
  *
  * Design constraints:
  * - The monitor runs out-of-band; it never blocks query execution.
+ * - Routing is applied **per-query**, not per-connection.
  * - Polling errors are treated conservatively: the replica is considered
  *   unhealthy until a successful measurement re-establishes a known-good state.
  * - The class is injectable (accepts a pool factory) so tests can supply a
@@ -242,6 +244,14 @@ export class ReplicaLagMonitor {
         lagMs,
         thresholdMs: this.lagThresholdMs,
       });
+      // Surface recovery so dashboards can distinguish "still breached" from
+      // "just restored" without relying solely on the lag gauge.
+      this.metrics.incrementCounter(
+        'db.replica.recovered',
+        undefined,
+        1,
+        'Number of times replica lag recovered below SLO and read routing resumed',
+      );
     }
   }