Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 44 additions & 138 deletions docs/horizon-history-gap-chaos.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Horizon Transaction-History Gap Injection and Detection

**Issue:** #518 β€” Stellar Horizon chaos: transaction-history gap injection and detection
**Issue:** [#706](https://github.com/RevoraOrg/Revora-Backend/issues/706) β€” Stellar Horizon chaos: transaction-history gap injection and detection
**Branch:** `test/horizon-history-gap-chaos`

---
Expand All @@ -27,29 +27,29 @@ This document describes:
```
HorizonTransactionHistoryFetcher
β”‚
β”œβ”€β”€ fetchPage callback (injected β€” real SDK or test double)
β”œβ”€β”€ fetchPage callback (injected β€” real SDK, createHorizonHttpFetchPage, or test double)
β”œβ”€β”€ Gap detection (BigInt paging-token arithmetic)
β”œβ”€β”€ Cursor state (last clean paging_token)
β”œβ”€β”€ Paused flag (set on fatal gap)
└── EventEmitter β†’ 'audit' channel
β”œβ”€β”€ ingest.page.ingested
β”œβ”€β”€ ingest.page.empty
β”œβ”€β”€ ingest.gap.detected
β”œβ”€β”€ ingest.cursor.paused ← fatal alert
β”œβ”€β”€ ingest.cursor.paused ← fatal alert (severity=fatal, alarm=horizon_history_gap)
└── ingest.cursor.resumed
```

---

## Files Changed
## Files

| Path | Purpose |
|------|---------|
| `src/services/horizonTransactionHistoryFetcher.ts` | Production ingest service |
| `src/services/__tests__/chaos/horizonGapChaos.test.ts` | Chaos test suite (11 describe blocks, 40+ tests) |
| `src/__tests__/mocks/horizonFake.ts` | Extended with `HorizonTransactionPageFake`, `GapSpec`, `seededRandom`, `buildDeterministicGaps` |
| `src/__tests__/fixtures/chaosHelpers.ts` | Extended with `runGapChaosScenario`, `buildSeededGapFake`, `filterAuditEvents` |
| `run-chaos-tests.sh` | Added `gap` scenario entry |
| `src/services/horizonTransactionHistoryFetcher.ts` | Production ingest service + `createHorizonHttpFetchPage` |
| `src/__tests__/chaos/horizonGapChaos.test.ts` | Chaos test suite (deterministic per seed) |
| `src/__tests__/mocks/horizonFake.ts` | `HorizonTransactionPageFake`, `GapSpec`, `seededRandom`, `buildDeterministicGaps` |
| `src/__tests__/fixtures/chaosHelpers.ts` | `runGapChaosScenario`, `buildSeededGapFake`, `filterAuditEvents` |
| `run-chaos-tests.sh` | `gap` scenario entry |
| `docs/horizon-history-gap-chaos.md` | This document |

---
Expand All @@ -60,152 +60,58 @@ HorizonTransactionHistoryFetcher
# Gap-injection scenario only (fast, deterministic)
./run-chaos-tests.sh gap

# Equivalent:
npx jest --testPathPattern="horizonGapChaos" --runInBand --forceExit

# All chaos tests
./run-chaos-tests.sh chaos

# Full suite with coverage
npm test -- --coverage
```

---

## HorizonTransactionHistoryFetcher

### Constructor

```typescript
new HorizonTransactionHistoryFetcher({
fetchPage: (cursor: string, limit: number) => Promise<HorizonTransactionPage>,
pageSize?: number, // 1–200, default 200
initialCursor?: string, // default "" (start from beginning)
haltOnGap?: boolean, // default true β€” safe mode
})
```

### Gap detection algorithm

Paging tokens are monotonically increasing 64-bit integers. After each page:

1. Compare `firstRecord.paging_token` to current `cursor`.
2. Parse both as `BigInt` (guarding against precision loss on large tokens).
3. If `firstToken > cursor + 1`, a gap exists β€” tokens between them are missing.
4. Scan within the page for intra-page gaps (same logic, consecutive records).
5. Return `GapDetail` on the first gap found.
## Acceptance criteria

Backward movement (`firstToken ≀ cursor`) is treated as a ledger reorg and is
**not** flagged as a gap β€” the page passes through.

### Halt and resume

When `haltOnGap: true` (default):

- Cursor is **not** advanced.
- `ingest.gap.detected` is emitted first, then `ingest.cursor.paused`.
- All subsequent `fetchNextPage()` calls return `{ paused: true }` without
calling the upstream `fetchPage` callback β€” the upstream is isolated.
- Recovery is explicit: the operator calls `resumeFromCursor(token)` to
acknowledge the gap and restart from a verified position.

When `haltOnGap: false`:

- Gap is reported via `ingest.gap.detected` only.
- Cursor advances to the last token on the page.
- Useful for monitoring-only setups that accept eventual completeness.
| Criterion | Behaviour |
|-----------|-----------|
| Gap injection is deterministic per seed | `seededRandom` / `buildDeterministicGaps` |
| Gap detection emits `ingest.cursor.paused` | Fatal alert; cursor does **not** advance |
| No silent skip of records | Halt returns empty `records` for the gapped page |
| Recovery when gaps close | Explicit `resumeFromCursor()` then clean pages |
| Multiple simultaneous gaps | Only the **first** gap halts; cursor stays at last clean token |

---

## Chaos Harness

### `HorizonTransactionPageFake`

A deterministic fake that:

- Generates sequential paging tokens starting from a configurable base.
- Accepts `GapSpec[]` β€” each spec names a token after which N tokens are skipped.
- Supports one-shot and permanent error injection for resilience testing.
## Wiring to Horizon

```typescript
const fake = new HorizonTransactionPageFake(1000, [
{ afterToken: '1004', skipCount: 3 }, // tokens 1005, 1006, 1007 will be missing
]);
import {
HorizonTransactionHistoryFetcher,
createHorizonHttpFetchPage,
} from './services/horizonTransactionHistoryFetcher';

const fetcher = new HorizonTransactionHistoryFetcher({
fetchPage: createHorizonHttpFetchPage(process.env.STELLAR_HORIZON_URL!),
});

fetcher.on('audit', (event) => {
if (event.type === 'ingest.cursor.paused') {
// page ops β€” severity is 'fatal'
}
});
```

### `buildDeterministicGaps(baseToken, totalPages, pageSize, gapCount, seed)`

Generates reproducible `GapSpec[]` using the mulberry32 PRNG. The same seed
always produces the same gap positions, making chaos scenarios stable across CI.

```typescript
const gaps = buildDeterministicGaps(1000, 10, 5, 2, /* seed */ 42);
// Always the same two gap positions for seed 42
```

### `runGapChaosScenario(pageFake, maxPages, haltOnGap, initialCursor)`

Drives a `HorizonTransactionHistoryFetcher` until it pauses (gap detected) or
exhausts all pages. Returns all audit events and page results for assertion.

---

## Test Suite Coverage

| Suite | What is tested |
|-------|----------------|
| `seededRandom` | PRNG determinism, range, different seeds |
| `buildDeterministicGaps` | Reproducibility, ordering, skip range |
| `HorizonTransactionPageFake` | Token sequence, gap skipping, error injection, reset |
| `HorizonTransactionHistoryFetcher – gap detection` | Clean pages, inter-page gaps, event ordering, cursor isolation |
| `haltOnGap=false` | Continue-through-gap mode, event suppression |
| `Recovery: resumeFromCursor()` | Resume after close, lastGapDetail cleared, idempotency, invalid input |
| `Multiple gaps – cursor isolation` | First gap halts; second gap never reached; haltOnGap=false reports once |
| `Deterministic chaos via buildSeededGapFake` | Same seed, different seeds, pause assertion, cursor bounds |
| `Gap recovery – gap closes` | Fresh fetcher from resume cursor, no spurious events |
| `Edge cases` | Empty page, single record, reorg tokens, non-numeric tokens, reset, large gap |
| `Multi-seed sweep` | 12 seeds; each must pause, emit one paused event, cursor within bounds |

---

## Security Assumptions

1. **No auto-skip.** The fetcher never silently skips gaps; `haltOnGap: true` is
the default. An operator action (`resumeFromCursor`) is required for
recovery, creating an explicit audit trail.

2. **Cursor immutability on gap.** The cursor is only written when records are
actually ingested. A gap page does not move the cursor, preventing phantom
advancement.
## Security assumptions

3. **Upstream isolation on pause.** When paused, the fetcher does not call the
upstream `fetchPage` callback β€” preventing repeated requests to a potentially
malicious or unstable upstream.

4. **BigInt precision.** Gap arithmetic uses `BigInt` throughout. JavaScript's
53-bit `Number` would silently lose precision on tokens > 2^53, which could
cause a real gap to appear as "no gap". `BigInt` eliminates this attack
surface.

5. **No raw upstream data in audit events.** Audit events carry structured
metadata (token strings, counts) β€” not raw HTTP responses. This prevents
server-controlled strings from polluting the audit log.

6. **Deterministic test seeds.** All chaos scenarios are seeded so CI runs are
reproducible. Non-deterministic randomness would allow flaky passes that
hide real gaps.
1. Horizon base URL comes from trusted config, never request input.
2. `paging_token` values are opaque; numeric ordering is for gap detection only.
3. Recovery is **explicit** β€” no automatic skip-over-gap.
4. BigInt arithmetic guards overflow on large tokens.

---

## Abuse and Failure Paths Validated

| Path | Test |
|------|------|
| Gap of 1 missing token | Suite 4 β€” inter-page gap, `missingCount: 2` (105,106) |
| Gap of 9 tokens (max skip) | `buildDeterministicGaps` edge, Suite 2 |
| Gap spanning >MAX_SAFE_INTEGER tokens | Suite 10 β€” very large gap clamped |
| Gap on very first page (no prior cursor) | Suite 10 β€” single record, no gap |
| Gap mid-page (intra-page) | Suite 7 β€” first gap detected within records |
| Two gaps on one page | Suite 7 β€” only first gap surfaces |
| Paused fetcher called again | Suite 4 β€” upstream not re-invoked |
| Upstream throws 503 | Suite 10 β€” error propagates correctly |
| Backward-moving token (reorg) | Suite 10 β€” not treated as a gap |
| Non-numeric token | Suite 10 β€” no crash, gap check skipped |
| 12-seed chaos sweep | Suite 11 β€” regression guard across seeds |
## Related

- `src/lib/stellarRpcClient.ts` β€” Stellar RPC client (submission / horizon health)
- Chaos siblings: `horizonChaos.test.ts`, `horizonBadSeqChaos.test.ts`
16 changes: 11 additions & 5 deletions run-chaos-tests.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
#!/bin/bash

# Run chaos tests with options
# Chaos test runner for Revora-Backend.
#
# Issue #706 β€” Horizon transaction-history gap-injection scenario:
# ./run-chaos-tests.sh gap
#
# The gap scenario is deterministic per seed (see buildDeterministicGaps /
# seededRandom) and asserts that gap detection emits `ingest.cursor.paused`
# (fatal) and halts cursor advancement rather than skipping records.

set -e

Expand Down Expand Up @@ -37,8 +43,8 @@ case "$1" in
npm run test:chaos
;;
gap)
echo "πŸ” Running Horizon transaction-history gap-injection chaos..."
npx jest --testPathPattern="horizonGapChaos" --runInBand --coverage
echo "πŸ” Running Horizon transaction-history gap-injection chaos (#706)..."
npx jest src/__tests__/chaos/horizonGapChaos.test.ts --runInBand --forceExit
;;
*)
echo "Running all tests..."
Expand All @@ -49,7 +55,7 @@ esac
echo "βœ… Tests completed!"
echo ""
echo "Available scenarios:"
echo " ./run-chaos-tests.sh gap – Horizon transaction-history gap-injection"
echo " ./run-chaos-tests.sh gap – Horizon transaction-history gap-injection (#706)"
echo " ./run-chaos-tests.sh chaos – All chaos tests"
echo " ./run-chaos-tests.sh coverage – Full suite with coverage report"
echo " ./run-chaos-tests.sh ci – CI mode (coverage + fail-fast)"
23 changes: 23 additions & 0 deletions src/__tests__/chaos/horizonGapChaos.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,29 @@ describe('HorizonTransactionHistoryFetcher – gap detection', () => {
expect(pausedEvents).toHaveLength(1);
expect(pausedEvents[0].cursor).toBe('104');
expect(pausedEvents[0].meta?.reason).toBe('gap_detected');
expect(pausedEvents[0].meta?.severity).toBe('fatal');
expect(pausedEvents[0].meta?.alarm).toBe('horizon_history_gap');
});

it('createHorizonHttpFetchPage builds a Horizon /transactions URL', async () => {
const { createHorizonHttpFetchPage } = await import(
'../../services/horizonTransactionHistoryFetcher'
);
const calls: string[] = [];
const fetchImpl = (async (input: string | URL) => {
calls.push(String(input));
return {
ok: true,
json: async () => makePage(['1', '2']),
} as Response;
}) as typeof fetch;

const fetchPage = createHorizonHttpFetchPage('https://horizon.test/', fetchImpl);
const page = await fetchPage('10', 5);
expect(page._embedded.records).toHaveLength(2);
expect(calls[0]).toContain('https://horizon.test/transactions');
expect(calls[0]).toContain('cursor=10');
expect(calls[0]).toContain('limit=5');
});

it('emits ingest.gap.detected before ingest.cursor.paused', async () => {
Expand Down
42 changes: 42 additions & 0 deletions src/services/horizonTransactionHistoryFetcher.ts
Original file line number Diff line number Diff line change
Expand Up @@ -230,8 +230,11 @@ export class HorizonTransactionHistoryFetcher extends EventEmitter {

if (this.haltOnGap) {
this.paused = true;
// Fatal alert: gap detection must halt advancement so no records are skipped.
this.emit('audit', this.buildAuditEvent('ingest.cursor.paused', {
reason: 'gap_detected',
severity: 'fatal',
alarm: 'horizon_history_gap',
...gapDetail,
}));

Expand Down Expand Up @@ -417,3 +420,42 @@ export class HorizonTransactionHistoryFetcher extends EventEmitter {
};
}
}

/**
* @notice Build a `fetchPage` callback that talks to a real Horizon HTTP endpoint.
*
* @dev Used to wire `HorizonTransactionHistoryFetcher` to Horizon (issue #706)
* without coupling the ingest layer to a specific SDK. The base URL must
* come from trusted config (e.g. `STELLAR_HORIZON_URL`), never from request
* input.
*
* @param horizonBaseUrl Horizon root URL, e.g. `https://horizon-testnet.stellar.org`
* @param fetchImpl Injectable fetch (defaults to global `fetch`)
*/
export function createHorizonHttpFetchPage(
horizonBaseUrl: string,
fetchImpl: typeof fetch = fetch,
): (cursor: string, limit: number) => Promise<HorizonTransactionPage> {
if (!horizonBaseUrl || typeof horizonBaseUrl !== 'string') {
throw new Error('createHorizonHttpFetchPage: horizonBaseUrl is required');
}
const base = horizonBaseUrl.replace(/\/+$/, '');

return async (cursor: string, limit: number): Promise<HorizonTransactionPage> => {
const url = new URL(`${base}/transactions`);
url.searchParams.set('order', 'asc');
url.searchParams.set('limit', String(Math.min(Math.max(limit, 1), 200)));
if (cursor) {
url.searchParams.set('cursor', cursor);
}

const res = await fetchImpl(url.toString(), {
method: 'GET',
headers: { Accept: 'application/json' },
});
if (!res.ok) {
throw new Error(`Horizon transactions fetch failed: HTTP ${res.status}`);
}
return (await res.json()) as HorizonTransactionPage;
};
}
Loading