diff --git a/docs/horizon-history-gap-chaos.md b/docs/horizon-history-gap-chaos.md index 26a62c3c..2ea59d2b 100644 --- a/docs/horizon-history-gap-chaos.md +++ b/docs/horizon-history-gap-chaos.md @@ -1,6 +1,6 @@ # Horizon Transaction-History Gap Injection and Detection -**Issue:** #518 — Stellar Horizon chaos: transaction-history gap injection and detection +**Issue:** [#706](https://github.com/RevoraOrg/Revora-Backend/issues/706) — Stellar Horizon chaos: transaction-history gap injection and detection **Branch:** `test/horizon-history-gap-chaos` --- @@ -27,7 +27,7 @@ This document describes: ``` HorizonTransactionHistoryFetcher │ -├── fetchPage callback (injected — real SDK or test double) +├── fetchPage callback (injected — real SDK, createHorizonHttpFetchPage, or test double) ├── Gap detection (BigInt paging-token arithmetic) ├── Cursor state (last clean paging_token) ├── Paused flag (set on fatal gap) @@ -35,21 +35,21 @@ HorizonTransactionHistoryFetcher ├── ingest.page.ingested ├── ingest.page.empty ├── ingest.gap.detected - ├── ingest.cursor.paused ← fatal alert + ├── ingest.cursor.paused ← fatal alert (severity=fatal, alarm=horizon_history_gap) └── ingest.cursor.resumed ``` --- -## Files Changed +## Files | Path | Purpose | |------|---------| -| `src/services/horizonTransactionHistoryFetcher.ts` | Production ingest service | -| `src/services/__tests__/chaos/horizonGapChaos.test.ts` | Chaos test suite (11 describe blocks, 40+ tests) | -| `src/__tests__/mocks/horizonFake.ts` | Extended with `HorizonTransactionPageFake`, `GapSpec`, `seededRandom`, `buildDeterministicGaps` | -| `src/__tests__/fixtures/chaosHelpers.ts` | Extended with `runGapChaosScenario`, `buildSeededGapFake`, `filterAuditEvents` | -| `run-chaos-tests.sh` | Added `gap` scenario entry | +| `src/services/horizonTransactionHistoryFetcher.ts` | Production ingest service + `createHorizonHttpFetchPage` | +| `src/__tests__/chaos/horizonGapChaos.test.ts` | Chaos test suite (deterministic per seed) | +| `src/__tests__/mocks/horizonFake.ts` | `HorizonTransactionPageFake`, `GapSpec`, `seededRandom`, `buildDeterministicGaps` | +| `src/__tests__/fixtures/chaosHelpers.ts` | `runGapChaosScenario`, `buildSeededGapFake`, `filterAuditEvents` | +| `run-chaos-tests.sh` | `gap` scenario entry | | `docs/horizon-history-gap-chaos.md` | This document | --- @@ -60,152 +60,58 @@ HorizonTransactionHistoryFetcher # Gap-injection scenario only (fast, deterministic) ./run-chaos-tests.sh gap +# Equivalent: +npx jest --testPathPattern="horizonGapChaos" --runInBand --forceExit + # All chaos tests ./run-chaos-tests.sh chaos - -# Full suite with coverage -npm test -- --coverage ``` --- -## HorizonTransactionHistoryFetcher - -### Constructor - -```typescript -new HorizonTransactionHistoryFetcher({ - fetchPage: (cursor: string, limit: number) => Promise, - pageSize?: number, // 1–200, default 200 - initialCursor?: string, // default "" (start from beginning) - haltOnGap?: boolean, // default true — safe mode -}) -``` - -### Gap detection algorithm - -Paging tokens are monotonically increasing 64-bit integers. After each page: - -1. Compare `firstRecord.paging_token` to current `cursor`. -2. Parse both as `BigInt` (guarding against precision loss on large tokens). -3. If `firstToken > cursor + 1`, a gap exists — tokens between them are missing. -4. Scan within the page for intra-page gaps (same logic, consecutive records). -5. Return `GapDetail` on the first gap found. +## Acceptance criteria -Backward movement (`firstToken ≤ cursor`) is treated as a ledger reorg and is -**not** flagged as a gap — the page passes through. - -### Halt and resume - -When `haltOnGap: true` (default): - -- Cursor is **not** advanced. -- `ingest.gap.detected` is emitted first, then `ingest.cursor.paused`. -- All subsequent `fetchNextPage()` calls return `{ paused: true }` without - calling the upstream `fetchPage` callback — the upstream is isolated. -- Recovery is explicit: the operator calls `resumeFromCursor(token)` to - acknowledge the gap and restart from a verified position. - -When `haltOnGap: false`: - -- Gap is reported via `ingest.gap.detected` only. -- Cursor advances to the last token on the page. -- Useful for monitoring-only setups that accept eventual completeness. +| Criterion | Behaviour | +|-----------|-----------| +| Gap injection is deterministic per seed | `seededRandom` / `buildDeterministicGaps` | +| Gap detection emits `ingest.cursor.paused` | Fatal alert; cursor does **not** advance | +| No silent skip of records | Halt returns empty `records` for the gapped page | +| Recovery when gaps close | Explicit `resumeFromCursor()` then clean pages | +| Multiple simultaneous gaps | Only the **first** gap halts; cursor stays at last clean token | --- -## Chaos Harness - -### `HorizonTransactionPageFake` - -A deterministic fake that: - -- Generates sequential paging tokens starting from a configurable base. -- Accepts `GapSpec[]` — each spec names a token after which N tokens are skipped. -- Supports one-shot and permanent error injection for resilience testing. +## Wiring to Horizon ```typescript -const fake = new HorizonTransactionPageFake(1000, [ - { afterToken: '1004', skipCount: 3 }, // tokens 1005, 1006, 1007 will be missing -]); +import { + HorizonTransactionHistoryFetcher, + createHorizonHttpFetchPage, +} from './services/horizonTransactionHistoryFetcher'; + +const fetcher = new HorizonTransactionHistoryFetcher({ + fetchPage: createHorizonHttpFetchPage(process.env.STELLAR_HORIZON_URL!), +}); + +fetcher.on('audit', (event) => { + if (event.type === 'ingest.cursor.paused') { + // page ops — severity is 'fatal' + } +}); ``` -### `buildDeterministicGaps(baseToken, totalPages, pageSize, gapCount, seed)` - -Generates reproducible `GapSpec[]` using the mulberry32 PRNG. The same seed -always produces the same gap positions, making chaos scenarios stable across CI. - -```typescript -const gaps = buildDeterministicGaps(1000, 10, 5, 2, /* seed */ 42); -// Always the same two gap positions for seed 42 -``` - -### `runGapChaosScenario(pageFake, maxPages, haltOnGap, initialCursor)` - -Drives a `HorizonTransactionHistoryFetcher` until it pauses (gap detected) or -exhausts all pages. Returns all audit events and page results for assertion. - ---- - -## Test Suite Coverage - -| Suite | What is tested | -|-------|----------------| -| `seededRandom` | PRNG determinism, range, different seeds | -| `buildDeterministicGaps` | Reproducibility, ordering, skip range | -| `HorizonTransactionPageFake` | Token sequence, gap skipping, error injection, reset | -| `HorizonTransactionHistoryFetcher – gap detection` | Clean pages, inter-page gaps, event ordering, cursor isolation | -| `haltOnGap=false` | Continue-through-gap mode, event suppression | -| `Recovery: resumeFromCursor()` | Resume after close, lastGapDetail cleared, idempotency, invalid input | -| `Multiple gaps – cursor isolation` | First gap halts; second gap never reached; haltOnGap=false reports once | -| `Deterministic chaos via buildSeededGapFake` | Same seed, different seeds, pause assertion, cursor bounds | -| `Gap recovery – gap closes` | Fresh fetcher from resume cursor, no spurious events | -| `Edge cases` | Empty page, single record, reorg tokens, non-numeric tokens, reset, large gap | -| `Multi-seed sweep` | 12 seeds; each must pause, emit one paused event, cursor within bounds | - --- -## Security Assumptions - -1. **No auto-skip.** The fetcher never silently skips gaps; `haltOnGap: true` is - the default. An operator action (`resumeFromCursor`) is required for - recovery, creating an explicit audit trail. - -2. **Cursor immutability on gap.** The cursor is only written when records are - actually ingested. A gap page does not move the cursor, preventing phantom - advancement. +## Security assumptions -3. **Upstream isolation on pause.** When paused, the fetcher does not call the - upstream `fetchPage` callback — preventing repeated requests to a potentially - malicious or unstable upstream. - -4. **BigInt precision.** Gap arithmetic uses `BigInt` throughout. JavaScript's - 53-bit `Number` would silently lose precision on tokens > 2^53, which could - cause a real gap to appear as "no gap". `BigInt` eliminates this attack - surface. - -5. **No raw upstream data in audit events.** Audit events carry structured - metadata (token strings, counts) — not raw HTTP responses. This prevents - server-controlled strings from polluting the audit log. - -6. **Deterministic test seeds.** All chaos scenarios are seeded so CI runs are - reproducible. Non-deterministic randomness would allow flaky passes that - hide real gaps. +1. Horizon base URL comes from trusted config, never request input. +2. `paging_token` values are opaque; numeric ordering is for gap detection only. +3. Recovery is **explicit** — no automatic skip-over-gap. +4. BigInt arithmetic guards overflow on large tokens. --- -## Abuse and Failure Paths Validated - -| Path | Test | -|------|------| -| Gap of 1 missing token | Suite 4 — inter-page gap, `missingCount: 2` (105,106) | -| Gap of 9 tokens (max skip) | `buildDeterministicGaps` edge, Suite 2 | -| Gap spanning >MAX_SAFE_INTEGER tokens | Suite 10 — very large gap clamped | -| Gap on very first page (no prior cursor) | Suite 10 — single record, no gap | -| Gap mid-page (intra-page) | Suite 7 — first gap detected within records | -| Two gaps on one page | Suite 7 — only first gap surfaces | -| Paused fetcher called again | Suite 4 — upstream not re-invoked | -| Upstream throws 503 | Suite 10 — error propagates correctly | -| Backward-moving token (reorg) | Suite 10 — not treated as a gap | -| Non-numeric token | Suite 10 — no crash, gap check skipped | -| 12-seed chaos sweep | Suite 11 — regression guard across seeds | +## Related + +- `src/lib/stellarRpcClient.ts` — Stellar RPC client (submission / horizon health) +- Chaos siblings: `horizonChaos.test.ts`, `horizonBadSeqChaos.test.ts` diff --git a/run-chaos-tests.sh b/run-chaos-tests.sh index 97cf2086..aff60b29 100755 --- a/run-chaos-tests.sh +++ b/run-chaos-tests.sh @@ -1,6 +1,12 @@ #!/bin/bash - -# Run chaos tests with options +# Chaos test runner for Revora-Backend. +# +# Issue #706 — Horizon transaction-history gap-injection scenario: +# ./run-chaos-tests.sh gap +# +# The gap scenario is deterministic per seed (see buildDeterministicGaps / +# seededRandom) and asserts that gap detection emits `ingest.cursor.paused` +# (fatal) and halts cursor advancement rather than skipping records. set -e @@ -37,8 +43,8 @@ case "$1" in npm run test:chaos ;; gap) - echo "🔍 Running Horizon transaction-history gap-injection chaos..." - npx jest --testPathPattern="horizonGapChaos" --runInBand --coverage + echo "🔍 Running Horizon transaction-history gap-injection chaos (#706)..." + npx jest src/__tests__/chaos/horizonGapChaos.test.ts --runInBand --forceExit ;; *) echo "Running all tests..." @@ -49,7 +55,7 @@ esac echo "✅ Tests completed!" echo "" echo "Available scenarios:" -echo " ./run-chaos-tests.sh gap – Horizon transaction-history gap-injection" +echo " ./run-chaos-tests.sh gap – Horizon transaction-history gap-injection (#706)" echo " ./run-chaos-tests.sh chaos – All chaos tests" echo " ./run-chaos-tests.sh coverage – Full suite with coverage report" echo " ./run-chaos-tests.sh ci – CI mode (coverage + fail-fast)" diff --git a/src/__tests__/chaos/horizonGapChaos.test.ts b/src/__tests__/chaos/horizonGapChaos.test.ts index 466b5154..8d9df8fe 100644 --- a/src/__tests__/chaos/horizonGapChaos.test.ts +++ b/src/__tests__/chaos/horizonGapChaos.test.ts @@ -254,6 +254,29 @@ describe('HorizonTransactionHistoryFetcher – gap detection', () => { expect(pausedEvents).toHaveLength(1); expect(pausedEvents[0].cursor).toBe('104'); expect(pausedEvents[0].meta?.reason).toBe('gap_detected'); + expect(pausedEvents[0].meta?.severity).toBe('fatal'); + expect(pausedEvents[0].meta?.alarm).toBe('horizon_history_gap'); + }); + + it('createHorizonHttpFetchPage builds a Horizon /transactions URL', async () => { + const { createHorizonHttpFetchPage } = await import( + '../../services/horizonTransactionHistoryFetcher' + ); + const calls: string[] = []; + const fetchImpl = (async (input: string | URL) => { + calls.push(String(input)); + return { + ok: true, + json: async () => makePage(['1', '2']), + } as Response; + }) as typeof fetch; + + const fetchPage = createHorizonHttpFetchPage('https://horizon.test/', fetchImpl); + const page = await fetchPage('10', 5); + expect(page._embedded.records).toHaveLength(2); + expect(calls[0]).toContain('https://horizon.test/transactions'); + expect(calls[0]).toContain('cursor=10'); + expect(calls[0]).toContain('limit=5'); }); it('emits ingest.gap.detected before ingest.cursor.paused', async () => { diff --git a/src/services/horizonTransactionHistoryFetcher.ts b/src/services/horizonTransactionHistoryFetcher.ts index 54afb80a..766efa6e 100644 --- a/src/services/horizonTransactionHistoryFetcher.ts +++ b/src/services/horizonTransactionHistoryFetcher.ts @@ -230,8 +230,11 @@ export class HorizonTransactionHistoryFetcher extends EventEmitter { if (this.haltOnGap) { this.paused = true; + // Fatal alert: gap detection must halt advancement so no records are skipped. this.emit('audit', this.buildAuditEvent('ingest.cursor.paused', { reason: 'gap_detected', + severity: 'fatal', + alarm: 'horizon_history_gap', ...gapDetail, })); @@ -417,3 +420,42 @@ export class HorizonTransactionHistoryFetcher extends EventEmitter { }; } } + +/** + * @notice Build a `fetchPage` callback that talks to a real Horizon HTTP endpoint. + * + * @dev Used to wire `HorizonTransactionHistoryFetcher` to Horizon (issue #706) + * without coupling the ingest layer to a specific SDK. The base URL must + * come from trusted config (e.g. `STELLAR_HORIZON_URL`), never from request + * input. + * + * @param horizonBaseUrl Horizon root URL, e.g. `https://horizon-testnet.stellar.org` + * @param fetchImpl Injectable fetch (defaults to global `fetch`) + */ +export function createHorizonHttpFetchPage( + horizonBaseUrl: string, + fetchImpl: typeof fetch = fetch, +): (cursor: string, limit: number) => Promise { + if (!horizonBaseUrl || typeof horizonBaseUrl !== 'string') { + throw new Error('createHorizonHttpFetchPage: horizonBaseUrl is required'); + } + const base = horizonBaseUrl.replace(/\/+$/, ''); + + return async (cursor: string, limit: number): Promise => { + const url = new URL(`${base}/transactions`); + url.searchParams.set('order', 'asc'); + url.searchParams.set('limit', String(Math.min(Math.max(limit, 1), 200))); + if (cursor) { + url.searchParams.set('cursor', cursor); + } + + const res = await fetchImpl(url.toString(), { + method: 'GET', + headers: { Accept: 'application/json' }, + }); + if (!res.ok) { + throw new Error(`Horizon transactions fetch failed: HTTP ${res.status}`); + } + return (await res.json()) as HorizonTransactionPage; + }; +}